Entries by Xan Gregg

The SAS Solar Farm data (available in the JMP File Exchange) has proven to be a rich topic for discussion and exploration. Besides the cool factor from green technology, the factors (such as sunlight, wind, temperature) can be understood by anyone, and yet the interactions are complex and not all linear. One issue that came up in the comments to my original post was the effect of ambient temperature on the power output. Rather than try to create an accurate model to account for sun position and solar panel angle, I tried some basic visual exploration to get a feel for the relationship of temperature and power.

A fair starting point is a basic plot of power versus temperature for the whole data set.



Here, I've set the graph transparency to 0.3 to give a point cloud effect. The visual is not too helpful except to get us to think about the other factors that are conditioning the relationship between power and temperature. Factors we have in our data set include time of day, day of year and solar irradiance. Other factors we might derive or get externally include solar angle, panel temperature and weather conditions.

Eschewing complex models, I tried conditioning the data on irradiance (sunlight) and time from solar noon (as a proxy for panel angle and sun position). The idea is that, say, two hours before noon and two hours after noon would have the same panel angle and sun position but likely different temperatures and power output levels. Solar noon, also called local apparent noon (LAN), is where the irradiance peaks on sunny days. We only have data in 15-minute intervals, and solar noon seems to be between 12:15 and 12:30 for the Cary, NC, area, and I chose 12:30 for my calculations. From that I calculated Minutes from LAN. Here's the power versus temperature conditioned on both Irradiance and Minutes from LAN.



Most of the panels show a slight negative relationship, as expected for solar cells. Eyeballing the trends suggests about 3kW per degree Celsius, or about 1% per degree. That seems a little high from what I've read, and I think it's because the panels still contain a bit of mixing of different conditions.

To go a step further, I decided to look at individual pairs of times equidistant from solar noon. With plenty of pairs to look at, I filtered it down to pairs with strong power output, a significant temperature difference (more than 4°C), a similar irradiance value and on a single array.



Each line connects a matched pair of temperature/power readings for a given day and panel angle (assuming the angle is proportional to minutes from local area noon). Now we can see that most pairs exhibit a small negative relationship, though there are a few outlier slopes in both directions. What accounts for those? Using Distribution or Tabulate, we find the median slope to be about -1.2 kW/°C, which is about 0.3% per degree based on an average 350kW base.

In my Solar Array Surprises post about the SAS solar farm, one of the surprises was a midday dip in the power output, for which commenters supplied several possible explanations.

That data was from April, and we could only speculate what the summer data would look like. But now summer is over (by some accounts), and we can look at eight months of solar output data.

Looking at the summer data brings a new surprise: It's very noisy.



I know we had a fair amount of rain this summer, but I remember some dry spells, too. I guess the clouds were always around. At least July 14 looks clear and sunny, which is better than I can say for any day in June or August.

Getting back to the midday dip, here's a Graph Builder plot using the Data Filter to show only the most sunny day in each month (hand-picked).



A few observations from the plot:

1. Power output is greater in the summer (height of the curve).


2. Daylight lasts longer in the summer (width of the curve).


3. The midday dip is more pronounced in the winter.


4. Summer output is noisier.

Points 1 and 2 are obvious. The third point supports the idea that the horizontal axis of the panel rotation is causing the dip, since it is more pronounced when the sun is lower in the sky. The fourth point could just be the weather, but it might be exacerbated by the power output being near capacity for the array.

Ambient temperature was also mentioned as a possible factor in the power output, but I haven't analyzed it. I was hoping to find at least one cool sunny day and one warm sunny day from each season for comparison. It's still possible the temperature accounts for the slightly higher output in the morning than the afternoon since solar cells are more efficient at lower temperatures.

UPDATE (09-09-09): The solar array data is now available from the JMP File Exchange. Scroll down to the bottom of my author page to find the file titled "Solar Array Data Jan - Aug 2009."

In my previous JSL tip, we saw how it is faster to access matrix items than to list items. It's faster still if you don't have to access the items at all. That is, get JMP's internal code to do the looping and item access instead of doing it in your JSL code. You can do that when you just need to apply a simple function to each item.

Recall the loop to make a (one-dimensional) matrix of odd numbers:

// make a matrix of odd numbers

n = 1000;

y = J(1, n);	// creates a 1 x n matrix with all values set to 1

for (i = 1, i <= n, i++,

	y[i] = 2 * i - 1;

);

Here's what it looks like without the loop:

// make a matrix of odd numbers

n = 1000;

y = 1::n;	// creates a 1 x n matrix with all values set to 1..n

y = 2 * y - 1;

Besides getting rid of the loop, we used a different matrix creation function. m::n is shorthand for Index(m, n), which creates a one-dimensional matrix with elements m to n, inclusive. When we apply scalar operations to a matrix, JMP applies those operations to each cell individually. That way, we transfer the looping from JSL to JMP internals. Almost any numeric function can be applied this way.

Sqrt( y );    // [1 1.41421356 1.7320508 2 2.23606 ...

Power( y, y ); // [1 4 27 256 ...

Log( y );     // 0 0.6931471805 1.09861228 1.38629436 ...

Floor( y / 3 ); // [0 0 1 1 1 2 2 2 3 3 ...

Sine( y / n ); // [0.0009999998 0.0019999986 0.0029999955 ...

As commenter Michael noted previously, the odd numbers example can be simplified even more because the Index function can actually take a third argument, which is an increment. So we can get the first n odd numbers with:

y = Index( 1, n * 2, 2 ); // odd numbers from 1 to 2n

The list is one of the basic building tools for organizing data in JSL. The power of lists is that they can contain any kind of items (numbers, characters, other lists, ...), and they can grow and shrink as needed. However, if you just need a fixed list of numbers, you can get much better performance from a one-dimensional matrix.

Consider these two samples that make arrays of odd numbers.

// make a list of odd numbers

n = 1000;

y = {};

for (i = 1, i <= n, i++,

	y[i] = 2 * i - 1;

);

y[n];

// make a matrix of odd numbers

n = 1000;

y = J(1, n);	// creates a 1 x n matrix with all values set to 1

for (i = 1, i <= n, i++,

	y[i] = 2 * i - 1;

);

For a few hundred items, there's not much difference. But as n goes beyond 1000 (or if the above code is in a loop), you start to notice the difference in speed. For lists, there is a certain amount of overhead for growing dynamically and general flexibility.

Notice the J function can be used to create a new matrix of a given size. J_m,n is the name for the Unit Matrix in linear algebra.

Also notice that a one-dimensional matrix can be indexed with just one subscript. That is, we can use y[i] instead of y[1][i]. That makes it easier to treat matrices as lists, and in the loop above, which is the same in both cases.

Question until next time: How can the code be made even faster?

Vector plots show arrows on a two-dimensional plot and allow one to see four dimensions of data: x position, y position, arrow angle, and arrow length. Equivalently, the four dimensions can be x start position, y start position, x end position, and y end position. The latter form is most convenient for JMP. Though JMP doesn't have a menu command to create vector plots, arrows can be added to almost any plot without much trouble.

While lots of four-variable data sets can be adapted for vector display, I'll demonstrate with one of the most natural applications: wind data. This data set records wind speed and direction for various weather stations in the Chicago area. I used formulas to convert the speed and direction into end positions for the arrows and then added the following graphics script to a bivariate scatterplot using Customize from the graph's right-click menu.

Pen Size( 2 );

For Each Row( Arrow( {lon1, lat1}, {lon2, lat2} ) );

Here's a screen capture showing a little bit of the data table, graph, and customize dialog:

The next steps were to delete the rows with no wind and to turn off the points in the graph.

To provide context, I added the map of weather stations provided by the weather service. Placing a image in a graph is a bit tricky (and could be the subject of another post). There are two options: Draw the picture in another custom graphics script, or drag and drop an existing image into the graph. I chose the latter since I had an image file. First, I sized the graph to be the same size as the image, and then I dragged the image file into the center of the graph. After a few tries, the alignment was pretty good. (BTW, we're working on making this better in JMP 9.)

The plot shows relative wind readings taken at 6 p.m. on May 31, 2009. With the help of the background image, we can see how wind patterns vary in relation to Lake Michigan.

UPDATE 10/6/09: The data file, with scripts, is now available in the JMP File Exchange. Look for the file called "ChicagoWind.jmp" in the Data Visualization section.

Source of data and image: the National Severe Storms Laboratory Historical Weather Data Archives, Norman, Oklahoma.

As you may have heard, SAS has a 1 megawatt solar farm in operation.

Naturally, I wanted to see what kind of data was available, and it turns out the power output is recorded every 15 minutes, even during the night (who knows -- maybe we'll get some power from the sunlight reflected by a full moon or from a passing comet). The output is measured separately for two halves of the farm called Array A and Array B. You can see the arrays, along with the sheep that maintain the grass at the solar farm, in the photo below.



Here's a graph of both arrays for the few sunny days in April we had here in Cary, North Carolina. I used the Data Filter to include only the sunny days and modified the axes to show only the daylight hours.



The flatness of the power output curve is a testament to the sun-tracking rotation of the panels. They're on a horizontal axis that runs north-south, and so can turn toward east or west.

Two interesting features stand out:

• There's a dip in the outputs around midday.
  
• Array A lags Array B in the early morning.

I think I know why these things are happening. But what do you think the reasons are?

In case you want to explore the solar farm data from April, it's available in the JMP File Exchange (scroll down to the bottom of the page to find the solar farm file).

SAS Solar Farm photo is by Dave Horne.

Some rare NFL scores, such as 11-10, have made the news recently (Wall Street Journal article), and resident sports expert Bob Hickey thought about looking at the scores in JMP.

Pro-Football-Reference has lots of views into the historical scores data but no one place we could find that supplied all the scores. After a bit of work, I managed to collect historical score data by aggregating data from many Web pages on that site.

The Wall Street Journal article mentioned that the modern scoring era started around 1978, so I limited my explorations to games since then. Here is a plot of all scores. Each score combination is represented by a square, with more common scores being darker.



The most popular score for this era is 20-17, same as for the entire history, but tie scores are now rare. The graph shows the expected dominance of scores like 13, 14, 17 and 20, which are basic combinations of touchdowns and field goals. It's also evident that close games are much more common than blow-outs.

If you're looking to predict a score for this weekend's Super Bowl, you might want to see the breakdown for playoff games only.



Scores of 20-17, 27-20 and 27-17 tie for the most occurrences at 6. There's not enough data to get a good sense of the distribution, but it's not noticeably dissimilar from the larger set of games.

I don't know who's going to win Super Bowl 43, but I'll predict the final score to be 27-17.

Arati Bechtel and I attended Science Online '09, which JMP sponsored for the third year running. The small size (capped at 200 attendees) and session format encouraged discussions rather than lectures, which helped everyone see from lots of perspectives. The issue of open access publishing was a central theme that came up in multiple guises, including various tools that have sprung up to help organize and link journal references and a new site that summarizes closed access research papers.

My favorite part was the stream of 15-minute demos on Sunday. I really just wanted to see Jean Claude Bradley's use of Second Life to visualize molecules, but getting short doses of different topics proved quite enthralling, and I stayed to hear the others.

The conference wiki includes a table of registrants and an Export button to get the data in various formats. It was nice to see the table open directly in JMP when I selected CSV (comma separated values) export. The data provides a good use for the Recode command, which let me easily clean the data for variations in spellings (such as "NC" versus "N.C." for the state column). Recode is a great feature, but there's always room for improvement -- what other data cleaning issues does JMP need to better address?

One of the discussion topics that generated a lot of interest in my Discovery 2008 conference session on Presentation Quality Graphics was the use of special effects such as adding a third dimension to two-dimensional plots.

Conventional wisdom in the data visualization community is that these effects add no value and often even distort the data being shown. Pie charts usually fall into that category, too. For an analysis on the effectiveness of pie charts, see Stephen Few's article Save the Pies for Dessert [PDF].

Nonetheless, some people love pie charts and special effects. In the conference discussion, these people were labeled "executives," and many engineers would rather please the executives than re-educate them.

JMP tends to focus on making graphs interactive and effective rather than pretty and eye-catching. JMP can do pie charts, but they are not a core visualization tool, and we have no plans of doing 3-D pie charts.

However, it occurred to me that JMP does provide a scriptable interface for 3-D graphs, and one could make a 3-D pie chart with a JSL script. After I put together a proof of concept, summer intern and Iowa State statistics student Jong-Seok Lee built it into to a usable script that prompts you for columns and generates 3-D pie charts. He added some nice features such as a grouping variable and the ability to make the pie sizes proportional to the group sizes.



And like any 3-D graph in JMP, you can grab each pie and rotate it to the view you want. Something to make any executive happy!

You can find the script at the bottom of this page in the JMP File Exchange.

Last week I showed how to make tornado charts in JMP and asked for input on the utility of these types of visualization. Here are thumbnails of the two alternative views of US population by age and sex.






One commenter pointed out that the back-to-back bars of the tornado style chart makes it easier to see the baby boom population bump and the smaller next generation bump 25 years later. While you can also see that general trend in the "normal" chart with side-by-side bars, it's not as obvious (especially the secondary bump) because the alternating gender values add choppiness to the trend curve.

Besides the trend of population versus age, the chart also shows the breakdown of population by gender for each age group. The back-to-back bars are good enough to see major variations, such as the larger proportion of females in the older age groups, but the side-by-side bars are better if you want to highlight smaller variations by gender. In the example, only the side-by-side bars make it clear that males are more common in the younger age groups.

The bottom line is that the appropriate visualization depends on the message you want to communicate.

During the Discovery 2008 conference, I heard a number of requests for tornado charts. Later, I realized a tornado chart is basically a horizontal, stacked bar chart with one negative variable and one positive variable, which is something JMP 7 can already handle. They're also called back-to-back charts, comparative histograms and population pyramids. Here's an example showing US population by age group:



Steps for creating this kind of plot in JMP:

1. Make sure there are two X variables (age and sex) and one Y variable (population). Tables Summary or Stack may be required to get the data in this form.


2. Make every other Y value negative (the females in the example).


3. Launch Chart with the two Xs and one Y and Horizontal orientation turned on.


4. Turn on Stack Bars within Chart.


5. Hide the divider lines by setting their color to off-white in Customize in the right-click menu.


6. Use the Grabber (hand) tool to move the legend into the graph.

So now that we can make tornado charts, what good are they? What kinds of data are they best for? How is the above tornado chart better or worse than the normal side-by-side bar chart (below)? Tell us what you think.

I didn't get any guesses on the odds of flipping various items from my Flipping Odds post. Instead, the commenters went directly to the question of interest, which is how does the chance for one flip affect the chance for two flips being the same? That turns out to be a straightforward formula. If we call the odds of landing up p, then the odds of landing the same on two flips turn out to be p*p + (1-p)*(1-p). That's a quadratic with a minimum of 0.5, so it's always better to call "same" if you're in the two-flip situation, such as at the start of an Ultimate Frisbee game.

Here's the JMP-made graph of the results that I showed in my Discovery 2008 session.



Notice that though the two-flip odds aren't 50%, they do get closer to 50% than each single-flip case. In the disc case, the odds go from 57% to 51%, so the double flip seems like a reasonable thing to do, even if it's not perfectly even odds.

For completeness, here are the numbers for landing in the "up" position. For the cat and the coin, I show theoretical values. The disc results represent 400 flips, and the others represent 100 flips each.

Object	Up Chance
Disc	57%
Lid	28%
Bowl	32%
Funnel	12%

I'm still not sure what the exact probabilities are, but JMP tells me they're not likely to be 50%. For instance, the probability of getting my disc results from a fair disc are less than 1%.

Statisticians like to talk about flipping coins, especially fair coins, which have an equal chance of being heads or tails. But what about flipping other things? In particular, as an Ultimate Frisbee player, I want to know what the odds are for a flying disc. Maybe knowing the odds will give me an advantage since a disc flip is used to determine which team starts on offense.

How do I figure out the answer? Flip a disc 10 times? 100 times? 1,000 times? Or study up on physics? If I flip a disc 100 times, what kind of answer do I need to be confident that the odds are not really 1:1 like a coin?

I'll tell you what I know in my Discovery 2008 session on Presentation Quality Graphics on June 16 at 4:15 p.m., but first tell me what you know. Post comments with theory and guesses on the chances of the following items landing as pictured. Two are filled in for you in the table below.

Bonus question: If I flip the same item twice, what are the odds of same versus different results?

Item	Picture	Chance
Coin		50%
Flying disc		?
Jar lid		?
Metal bowl		?
Funnel		?
Cat		100%

The 2008 NC Science Blogging Conference was held last week at the Sigma Xi center in nearby Research Triangle Park. This was the second year of the conference, and JMP has been a sponsor both years. Besides bloggers, of course, the audience of 200 included scientists and teachers.

This year I co-led a session called "Public Scientific Data" with Jean-Claude Bradley from Drexel. Jean-Claude already has the ultimate position on public data, with every experiment and result from his UsefulChem malaria lab going onto his website. My interest is in data visualization, and I'm often stymied by a lack of public access to data when I try to improve a visualization I see in a journal. The session was successful in raising awareness in some, at least, on the need and techniques for sharing data, though I found out that competition among researchers is a big obstacle to sharing.

My slides are available in HTML, and Jean-Claude has posted the audio and screencast recordings on his blog.

The most common question we get regarding Stephen Few's white paper and webcast on visualizing change is about the scripts for showing the connected trails in bubble plots. Often the existing bubble trails are overlapping, so the progression is clear, but when the bubble trails are spaced out it can be helpful to connect the bubbles with a line.



In addition to the journal file that contains the data and plots from the paper, you can also download from the JMP Extras area the graphics script by itself which can be added to any bubble plot (via Right-Click and Customize) if you edit the column names in the script.

Local( {s = "", c = 0, xx = {}, yy = {}},


  For Each Row(


    If( Selected( Row State() ),


      s = :State;


      c = Color Of( Row State() );


      Insert Into( xx, :Property Rate );


      Insert Into( yy, :Violent Rate );


    );


    If( N Items( xx ) != 0 & (Row() == N Rows()


        | !Selected( Row State( Row() + 1 ) )


        | s != :State[Row() + 1]),


      Pen Color( c );


      Pen Size( 2 );


      Transparency( 0.3 );


      Line( Matrix( xx ), Matrix( yy ) );


      xx = {};


      yy = {};


    );


  )


);

The script requires that the data be sorted by the ID column, "State" in this case.