The Demise of the Gansy of Gnomes Gardening Club

This is a tale of woe and misery, but also one of hope.

Woe and misery

dark volcano

Our tale begins more than seven years ago, an age ago in Internet Time. A cloud (nay, another type of cloud than what may come to your mind in this Age) pervaded the halls of SAS' R&D division. Our internal R&D content, not unlike the desolation that lay before Mordor, reeked, "as if the mountains had vomited the filth of their entrails of documentation upon the intranet about." While there was some centralized documentation (lo, over a million scrolls) entombed in a behemoth named Rdweb, each division, many departments, and even individuals ran their own web servers, writ with current reference and lore, interspersed with with obsolete and abandoned scrolls, and no one was sage enough to know which was which. Content was locked away in dungeons guarded by wargs, and only a few magicians knew the incantations to unlock the barriers required to cleanse and repair a tattered scroll or burn a scroll beset with lesions of deceit and trickery. When the Plague of Reorg swept across the land, old names were cast asunder, new names were forged, web servers were cleft, and Favorites, Links, and Shortcuts were rendered useless. A fell wind was upon the land.

Hark! for a new wizardry was growing in the land, one born forth in the fifth year of the millenium, sprung forth from the land called MediaWiki, and a new greenness emerged upon the meadows of Research and the vales of Development, and it was called saspedia. Its intendment was to be the common font of knowledge for how SAS R&D masters of wares both soft and firm could record their efforts for generations to come: an enduring and lasting home which would remain long after lordships and kingdoms rose and fell. Under the guiding hand of the WWW—White Wizard of Wiki—scribes could repair the Scars of Typos, the Fever of Incorrectness, and the dreaded Links of Red. In that first year, almost 600 new virtual scolls were writ; in the next, another 1,200, and saspedia has since increased in girth and growth, from 1 millipedia (0.1% the number of scrolls of Great Wikipedia) to 5 millipedia in the seventh years of the Era.

Alas, the weeds still grew and threaten to choke the fertile grain. Many pages were created, then abandoned. An invasion of Transient Content, culled from the Mireful Messages of Email Exchange, gained purchase in saspedia, though they fit its intendment not well. Some scrolls were tattered on the edges; lacked tendrils connecting them to related scrolls; many had appellations misleading; new and inconsistent terms were as rampant as mice in a moor; few scrolls were added to the Grand Scheme of Content Categorization. A band of weary yet merry folk, unlike Hobbits and unlike Dwarves, and fairer of face, emerged to tend the garden.  Being strong of heart, they banded together and formed the Gansy of Gnomes Gardening Club, and endeavored to teach others the attainable craft Wiki Markup, the lore of Content Categorization, the satisfying magic of Attribution, and the alchemy of turning Links of Red into Links of Blue. Their greencraft established the four pillars of saspedia scribes: Context, Categories, Content, and Contacts, and word spread, though slowly.

Woe, for the readership and writership of saspedia was small, unlike the millions who sustain and fertilize Great Wikipedia Still, they continued, and christened journals to record their work, both completed and enqueued, and sought more to join their Gansy. Soon, the great Eye of Doc was consulted, and form and structure came to saspedia in the form and substance of Semantic Wiki, and with the blessings of kings and lords. Some joined the Gansy of Gnomes Gardening Club, but their tenure was short, and the gnomes were overcome by a raging Flood of Content. The Gansy sensed that their fellowship was breaking, and with time, they sought separate quests.

Hope

Tulips: a sign of new growth

saspedia continues to grow. Individual gnomes contribute where they can. Some 20,000 scrolls have been penned by this twelfth year of the millenium, the seventh in the Era of saspedia. The original canons still apply, for saspedia is firm and immune to the Plague of Reorg, which has returned several o'er and o'er to scour the land and make way for rebirth of Improved Processes. Some lordships and kingdoms remain, for the Kingdom of SharePoint also arose in the land at the same time, and the threat of Opaqueness of Word of Redmond remains to this day.

But the White Wizard of Wiki is wise, and more and more adherents adopt that Wiki Way, for it indeed is exemplary in its Ease of Use: the  Gansy of Gnomes left behind Guide Books and Templates to make the work light and quick. saspedia serves all, whether they wield of Sword of Windows, the Hammer of MacOS, or the Lance of Linux. The White Wizard of Wiki continues to ride forth on his inspiring steed, Collaboration. Though the Gansy no longer flourishes and is more memory than flesh, their legacy lives, and hope remains where heart is true.

 


Epilogue

An award (including a real plaster gardening gnome) given to saspedia Wiki Gnomes of note

The author created saspedia, SAS' internal wiki, in 2005 to address the problems of R&D's internal documentation.  saspedia found executive sponsors and now is the official medium for authoring and collaboration on internal documentation, including internal development standards, processes, tools, products, infrastructure, teams and individuals, and much more. A project named iDoc formalized and facilitated cross-divisional collaboration on saspedia, using Semantic MediaWiki to classify content types (reference, How-to, requirements, plans, project logs, samples, FAQs, etc.), achitectural tiers, status, ownership and when the content was last reviewed.

There are over 5,000 registered peers, 20,000 main namespace articles, over 52,000 total pages, 585,000 page edits (averaging 11+ crowdsourcing edits per page), and 17+ million page views as of the date of this article's publication. saspedia's scope has since grown beyond just R&D. The Gansy of Gnomes Gardening Club is real; the author founded it as a non-threatening way to entice others to adopt the Wiki Way. The award at the right (including a real plaster gardening gnome) was given to saspedia Wiki Gnomes of note. But the idea never caught on and the Club has fallen into demise. Although the infrastructure (servers, database, etc.) is supported by our outstanding intranet support team, saspedia is still maintained and tended by volunteer gnomes, not a dedicated staff. It could benefit from dedicated staff who could create training material, assist new users acquire wiki markup skills, develop domain-specific  templates, remove duplication, enhance connectedness, etc. Such is life in a large R&D organization where innovation and inventiveness is prized. Yet saspedia flourishes and the author's peers find new ways to use it all the time.

The author speaks on the use of saspedia and continues to serve as advocate and, with a small number of others, as wiki gnome.


 

Photo credits:
Volcano derived from Volcan Popocatepetl by M. Klüber Fotografie under the Creative Commons Attribution-Share Alike 3.0 Unported license;
Good morning, tulips  by the author

Post a Comment

Rock stars

“They are a group of SAS rock stars and I'm darn proud of them!”

That comes from Meg Pounds, manager of the Customer Experience Testing (CET) team. (Go on, Meg - use some stronger language; Peer Revue needs some juicier content.)

birthday cake

CET celebrates its 10th anniversary on 02/01/2011. (Photo by Meg Pounds)

Meg's remark comes on the occasion of the team's tenth anniversary of testing SAS software. CET's motto is “Act Like a Customer” (ALAC). These folks install and use SAS the way our customers do, but do so while the software is still under development.

Not part of any R&D development division, they can maintain impartiality and loyalty to the customers they represent.

This team of a baker's dozen focuses on finding defects that customers would otherwise encounter. This often involves using SAS in ways not originally envisioned by the development teams, or working with different combinations of products in non-standard, non-default deployment scenarios, and seeing how well they integrate. When it comes to SAS product installation and deployment wizards, these folks never click Next, Next, Next. Instead, they tweak all the options, put things in non-default locations, etc. Just like a customer.

Typical comments about CET and the staff from development managers run like this (paraphrased):


CET found an error in the updateConfigure code for the OpRisk Monitor middle tier that would have
impacted a lot of customers. Both CET members are very good at identifying and giving me feedback
how to best resolve the errors.

This type of “experience with experience” should not be surprising: CET staff members have from 12 to 24 years with SAS, averaging over 18. (That kind of tenure is not an outlier at SAS, but is one of the "benefits of benefits".) Many CETers are veterans of SAS' awesome Technical Support division and SAS field teams.

Customer experience testing is a great model for testing which complements other forms of testing we do at SAS. Deep down, there is simply a lot of joy and satisfaction in finding a problem early and seeing the positive impact it makes on product quality. In addition to finding software problems. For example, Meg attributes CET use and review of SAS documentation as a contributing to the multiple quality awards that SAS documentation has received in recent years. Keep up the good work, CET. I'm looking forward to a few more decades of success.

Post a Comment

Perks are epic

When I started Peer Revue, I tried to point out it's the work we do that makes SAS such a cool place to be, not just the fancy-schmancy benefits that get all the headlines. But FORTUNE magazine apparently has another agenda; they've gone and named SAS the Best Company to Work For in America again (we had the same honor last year), stating “perks are epic”. I thought, Oh, no! Yet another story about SAS' great benefits!

1+1=1, or The True Meaning of SAS' RETAIN Statement

#1 last year plus #1 this year equals #1


The latest news comes down to this: SAS is a seriously epic great place to work. My peers and I know this very well. Personally, all the perks and policies do exactly what they are designed for: keeping employees happy, motivated, and productive. We like coming to work each day, and we have far fewer distractions when we're here.

Example #1.
I went to SAS' on-site Health Care Center for my annual physical on Tuesday. This curmudgeon is approaching a half century on this mortal coil, and I should be getting
regular physicals. At SAS, it's easy to do. I left the office at 8:20 for an 8:30 appointment; I was in with the nurse by 8:32 and back to work faster than you can snap your latex glove. Elsewhere, I can see that it would be easy to not go in for poking and probing each year. Here's another cool thing you probably have not heard
in the press: the perks result in high retention, and not just for folks like me, but for everyone at SAS, including for the health care staff. I've been seeing the same doctor and nurse practitioner on-site for years.

Example #1 ('cuz everything feels like #1 today)
My peer, Caroline Brickley, also has a compelling story about how SAS' benefits greatly reduced her anxieties about an expensive and potentially stressful child adoption. I hope you'll read it.

Here's a final Example #1, another adoption story.
This month, I got a desktop PC upgrade—I've had my more “mature” PC long enough to meet the upgrade eligibility requirements. One of my peers on the SAS IT staff configured a new PC with all the default installed software, and with literally a push of a button, it restored my personal settings, documents, and data from automatic backups. I had all the time I needed to install the additional development software (Eclipse, Java 6, GNU Emacs, etc.) that I use. It was easy to adopt this new PC because the IT staff has the resources and skills to make this process fast and easy. (Fear not; IT will find a suitable green pasture where the obsolete PC can spend the rest of its mortal coil).

All that, and more, adds up to a work environment that lets me and my peers get our jobs done with fewer distractions, less time away, and less stress. That means we can remain more focused on delivering better software to our customers.

#1 last year, #1 this year. Add 'em up; it does not change: we're still #1.

Post a Comment

Spirits in the virtual world

As the holiday season approaches, one is tempted to use a Dickensian reference
about spirits past, present and future.
But instead of a holiday classic from 1843, I'm going to haunt you with an earworm from 1981.
Having matriculated in the same year that The Police recorded Spirits in the Material World on their Ghosts in the Machine album, the song is deeply embedded in my brain.
It has begun popping up in my mind more and more, for I consider it a theme song for the material world counter-cultural phenomenon,
virtualization. This leads us to RACE, the epitome of virtual worlds here at SAS.

Like iTunes for SAS applications

RACE is SAS' Remote Access Computing Environment—a growing collection of virtual machines in a cloud computing environment.
Whoa, OK, lots of buzzwords there, so I'll break it down. A few years ago, the friendly folks going on the road for SAS needed to demonstrate
complex SAS solutions which ran on multiple tiers: desktops running browsers, SAS workspace servers running SAS jobs, a SAS Metadata server, SAS/Share server, an application server running SAS the middle-tier platform, etc. This did not all fit well in a road warrior's laptop.

To help these folks, SAS' information technology teams assembled a remote access system that these weary travelers could connect to. But there were lots of different configured systems, not all of which were needed each day, and not enough servers to keep
hundreds of different but infrequently deployed environments sitting around mostly unused. Rather than installing and configuring new systems each time someone needed to do a new demo, the IT folks chose virtual machine technology to host these SAS deployments. Someone could start with a clean system running on top of a virtual machine hosting environment, install and configure some SAS applications, load demo data, then click a button to take a "snapshot" of that system. Then, to save the time and effort to repeat that configuration, they could put that snapshot in a repository. When needed, the road warriors could pick up that demo image and run it, or they could run a different image with a different set of products or demo data. These deployments became reusable components in a library, not unlike various Police MP3 files in your digital audio library. Feel like listening to Spirits in the Material World or another classic Police earworm, Don't Stand So Close to Me? Simply search for it, load it up, and listen. Feel like running SAS with SAS Campaign Studio, SAS Web Analytics, and a SAS Business Intelligence server stack all rolled together? Locate the configured SAS software image from the RACE virtual machine library, reserve a server for it from the pool of RACE machines, and run it.

Breakdown of different SAS division's use of RACE images on December 9, 2010.

Davy Rowland, IT Manager,
describes RACE: "At the heart of the RACE Cloud is an internally developed software application that manages CPU, memory, network and data storage resources and provides self-service IAAS (infrastructure as a service)... functionality to 11,500 employees globally. The Scheduling and Image Management System (SIMS) orchestrates the construction and deployment of servers within the cloud. The cloud architecture uses Network Appliance storage accessible via SAN as the foundation of the cloud for near-instantaneous cloning of disk images."
Naturally, this idea caught on at SAS and in 2009, R&D joined RACE, using RACE images for testing.
Davy wrote, "IT and R&D recognized that product testing groups could benefit from the repeatable and reusable capabilities of the RACE Cloud image library. A small group of developers build library images that are shared and used by all the testers, resulting in faster and more robust testing cycles. Software testers spend more time testing SAS products rather than configuring the latest version of the products on their machines". The project became known as RACE 4R&D.

Test more, Test more often

Clearly, RACE is a boon to testers. Says Edie Jeffreys, Development Tester,
"What I love about RACE is the ability to quickly start with a bank of 'clean machines' to host my deployments and the ability to put up multiple similar multi-tier deployments and compare differences when I introduce different circumstances in each of them. I can easily build out multi-tier configurations including distributed web tiers". Sharon Stanners, Senior Manager, Software Development, was part of the team that worked with Davy on Race 4R&D. Sharon says "We formed a very close virtual team [Ha! Good one, Sharon!] and we extended the team to include some senior developers in R&D. We fixed a few things to make RACE more useful for Ramp;D including larger default images, more default RAM, access to [our network], extended default reservation times, and NetBIOS and DNS names that match".

On average, the RACE system delivers a fully operational, server-grade computing resource in less than 10 minutes.
In December 9, 2010 alone, there were 919 active system reservations from R&D, more than half of the RACE reservations that day.
This has increased our ability to test more often, because testers spend far less time setting up systems. It's trivial to reset a test system to a known state because testers just have to restart the virtual machine image. There is no delay to uninstall or reinstall and reconfigure products.

Community-based testing

Recently, Dr. AnnMaria De Mars (@annmariastat on Twitter) wondered why "SAS doesn't rely on their user community more" for testing new releases. Dr. De Mars wrote in Hybrid Open-Source SAS � or what are friends for?:

You would think that somewhere in those million users you would have people who would be happy to dive in and break things (also known as testing).

With a large enough user base, SAS could make copies of work in progress available for testing which would allow identification of problems. I can see many reasons that users would be happy to do this:

  • Consultants would see this as an opportunity to get ahead of the curve, using the newest software before it was available to the general public.
  • Students could use this chance to learn more about the latest software.
  • Just for the hell of it. (This is my motivation for most things.)

I like to call this community-based testing; others might classify this as crowdsourcing.
Certainly, SAS loves its customers. Perhaps we love them too much to let them
serve as testers!
Seriously, though, SAS already does work with some customers as early adopters
and development partners. But is there room for more?

Cookie cutters

One of the obstacles to supporting such community-based testing is the complexity of a
typical SAS installation.
Of course, the type of community-based testing that RACE could support initially might be limited to
functional and performance testing: Does this SAS procedure do what I expect?
Does it perform as fast on my data as the previous release?
Testing different deployment scenarios—a tougher testing problem; imagine all the different permutations—might
be beyond the scope of preconfigured RACE images.
Meg Pounds, who runs our Customer Experience Testing (CET) team,
knows this all too well, stating. "I encourage my team to use 'real' machines whenever possible."
That is, our customer's typically do not run SAS on preconfigured, nicely boxed systems, or as Meg calls
them, "'cookie cutter' images."
Meg's CET team ("a very fine group of highly skilled testers...! Wish we had more!")
has a mission to "Act-Like-A-Customer." (I am fairly certain that CET's mascot is not a duck shouting
"ALAC! ALAC!" in a series of television commercials.)
Who better to act like a customer than a real customer?
However, the many difficulties of making public test images of SAS products and solutions available for our customer
base are beyond the scope of this post (read: "it's too much work for me").
So, while this blogger cannot say that SAS will act on Dr. De Mar's suggestion (and Peer Revue is definitely
not announcing any such plans!), the idea is getting some review.
Whether implemented as suggested or not, it may alter our development and testing processes for the better.

Steal this idea

SAS' RACE technology has been a watershed for R&D.
My peers in testing have found great productivity gains.
Perhaps we will find a good way of using something like RACE to let our great customers provide more early feedback
on SAS products and solutions—a sort of special preview concert event for loyal listeners.

Information Week recognized RACE as one of "20 Great Ideas To Steal" in 2010. But don't steal a Police MP3; it will haunt your mind.

Post a Comment

SAS usability gets peer review

Like a car loan officer hot on the heels of usability car salesmen, and
in commemoration of World Usability Day,
I'd like to list some of the papers and posters which my peers have presented at recent usability conferences.
This will give Peer Revue readers a sense of the type of peer reviewed work my SAS peers have done to advance just one of the many fields related to SAS product development.

  1. Leslie Tudor, SAS,
    Transforming the Role of Usability in a Corporate Environment: Case Study for Usability Innovators
    Usability Professionals' Association 2011 (submitted)
  2. Leslie Tudor, Rajiv Ramarajan, Huifang Wang, SAS,
    Usability Group Management Methods: How to Create a Cohesive, Innovative, and HighPerforming Team
    Applied Human Factors and Ergonomics Conference, 2010
  3. Paulo Santos, Alcatel-Lucent, Cheryl L. Coyle, SAS,
    Investigating Query Schemes with Mobile Users Across the Globe (Paper and Poster),
    International Association for Development of the Information Society 2010
  4. Excerpt from the Orthographic TreeMap poster by Benson et. al. for InfoVis 2010. Click to view the entire poster.
    Excerpt from the Orthographic TreeMap poster by Benson et. al. for InfoVis 2010. Click to view the entire poster.

    Jordan Riley Benson, Lee Sullvan, Rajiv Ramarajan, Frank Wimmer, Paul Hankey, SAS,
    Using Orthographic Projection and Animation to Convey Treemap Structure (Poster),
    IEEE Information Visualization Conference (IEEE InfoVis) 2010

  5. Lisa Whitman, NCSU (now at SAS),
    Behavior and Common Mistakes of Novice Computer Users: An Evaluation of Errors Committed by Students Learning Windows
    Proceedings of HCI International 2009 International Conference on Human-Computer Interaction 2009
  6. Lisa Whitman, NCSU (now at SAS),
    The Effectiveness of Interactivity in Computer-Based Instructional Diagrams,
    Proceedings of HCI International 2009 International Conference on Human-Computer Interaction 2009
  7. Michele Vanvolkom, Monmouth University,
    Janice Stapley, Monmouth University,
    Heather Vaughn, Alcatel Lucent,
    Cheryl L. Coyle, Alcatel Lucent, SAS;
    Age and Gender Differences in Communication Technology Use,
    Eastern Psychological Association 2009
  8. Paulo Santos, Alcatel Lucent
    Cheryl Coyle, Alcatel Lucent, SAS
    Heather Vaughn, Alcatel Lucent
    How�to�Run�a�Global�Study�with�Mobile�Users;
    Proceedings of HCI International 2009 International Conference on Human-Computer Interaction 2009

Show me

Let's look under the hood at one of these papers, chosen after painstaking research and some weighted fuzzy dice:
Using Orthographic Projection and Animation to Convey Treemap Structure by Benson et. al.

A few years ago, Lee Sullivan started experimenting with the idea of overcoming some of the usability problems
of a treemap view of hierarchical data and developed a proof of concept animation prototype using 3D orthographic projections.
More recently, Riley Benson moved that work forward and implemented those and other ideas in Flex,
sharing the prototypes in an iterative development/rapid prototyping methodology with the team
and later with a wider internal audience. Those peers included developers who used a traditional
two-dimensional tree map in their product. Their review helped close the feature gaps between what was
expected and what Riley's new implementation provided. Testing also included run in
SAS' usability lab, with the assistance of Paul Hankey.


Take a look at the demonstration video on YouTube:

Riley says of the process,
the potentially interesting part of our method for developing the orthographic tree map is that we didn't start from a problem and work toward a solution tailored to solve it. We started with an idea, filled it in until we felt that it was well defined, and then began to experiment with the application of the idea to find what types of problems it was best suited to solving. I think this greatly contributed to the creative environment that has surrounded this project since it began.

While the new Orthographic TreeMap is not yet a component in SAS products,
it does demonstrate the spirit of innovation and moving forward that I find so rewarding at SAS.

Follow this bit.ly bundle link for a breakdown of the links included in this blog and a preview of each cited article or web site.
Post a Comment

Usability car salesmen

Hot on the wake of World Statistics Day, when there were close to 7,012 blogs from SAS about it,
World Usability Day almost passed unnoticed. But the keen-eyed noted that at SAS, we celebrated World Usability Day
with an internal showcase event.

Between 11:00am and 2:00pm, my peers showed off the all the work that they've been
doing to make SAS products easier to use—you know, "making life easier."
There were numerous demo stations, some games demonstrating how usability
analysts use "task analysis" to learn how people perform work, or how
eye trackers tell us how people really perceive a user interface.
Some of us submitted usability stories to get in a drawing to win an Amazon Kindle.
(My odds were good, but I don't think I won...)

BumbleFly

One of my favorite demo stations at our usability fair was manned by my peers,
Jeff Diamond and Rajendra Singh, who work in the
Component Design and Development (CDD) department,
creating enhanced Adobe Flex
user interface components. In order to promote reuse and consistency
across SAS web applications, it's important to get everyone on the same
page, and in this case, that page is a sample application
called BumbleFly.

BumbleFly brilliantly showcases all of the Flex UI controls.
It lists them alphabetically or by category, or you can search by component name or keyword
like "list" or "chart" or "box".
When you select a component,
BumbleFly presents a working sample in the main panel, where
you can interact with the component and try its features.
Tabs at the bottom
contain sample MXML source used to embed the control,
API reference documentation, internal usage and support documentation (on saspedia—I'll blog about saspedia soon), SAS usability guidelines related
to the component, and/or any open defects.

Here is what BumbleFly looks like when viewing the DecompTreeChart.
(Click on the image to view a full-resolution version.)

BumbleFly demonstrating the DecompTreeChart component

Of course, the BumbleFly application is a Flex application implemented with
all the CDD controls. But the implementation is only half the story.
I don't call Jeff and Raj "usability car salesmen" in a negative sense.
Rather, when you need a new car, they are like the salesmen who help you find exactly what you need
because they are also the guys who built the cars.
What's way cool about BumbleFly is that it bundles (rather than bumbles!) everything
together in one place, making it fast and easy to find and assemble what you need into
an application. It solves a classic reuse dilemma—how to you document
and classify the components that are available for reuse? It capitalizes
on our visual sense and recognition skills, and thus is highly usable.
Not a bad way to celebrate World Usability Day.

Post a Comment

I <3 STATISTICS

Like John Sall, I'm wearing my "I <3 STATISTICS" pin today... and I'm not even a statistician.

When I tell new acquaintances that I work at SAS, I usually get one of these responses:

  • “Oh that's nice. I've never heard of them.”
  • “Oh that's nice. I hear they have great benefits. So, what does SAS do?”
  • “Oh that's nice. I used SAS in college. So, you're a statistician, right?

Nope, I'm not a statistician. I don't even play one on TV. I did take statistics class in college. One class. In 1984. To me, box and whiskers is a cute picture my daughter took of one of the family felines.

Quite a few of my peers develop SAS products but don't do statistics, either. One product development manager says "Approximately 6% of our code involves statistical/analytical calls" while another claims of his team, "I count 15/40 or 37.5% of the [...] developers who write SAS-based analytics code." (I think the margin of error on that is +/- 5%, but don't quote me on that—I'm not a statistician.) But this low rate of statistical prowess should not be a surprise when one looks at the complexities of today's enterprise solutions, which need sophisticated user interfaces, security, data and metadata management, workflow, scalability, etc.

This does not mean that statistics and analytics are not crucially important to SAS products such as SAS Web Analytics and SAS Social Media Analytics and many, many more. I don't know how a disk controller works, how a relational data base and SQL processor works, how to render a 3D shaded surface map... and I don't know how SAS statistics and analytics work. (Instead, I know how a Java virtual machine works, how to design APIs, how to write (and how not to write) a concurrent program, how an LL(1) parser works, and how map and fold and reduce (and a little bit of monads) work in functional languages.) We all have our areas of expertise and clean separation of concerns, and that is what enables SAS to develop such a wide variety of products and solutions. More crucially, we know how to reuse rather than reinvent the features we need—including statistics—when assembling a software solution.

Luckily, my peers know how statistics work, and the strength of those methods is what sets our solutions apart.  Thus, even though we may not all understand how they work, WE <3 STATISTICS. So, thanks for the pin, John, and let's all celebrate World Statistics Day.

Post a Comment

Communication is easy

Communication is easy? Wishful thinking?

Keith Collins, our CTO and VP of Technology, once told me "communication is hard." That may seem a completely obvious statement to most of Peer Revue's readers, but recall I'm a code monkey. We developers think communication is easy:

we write a program that tells the computer exactly what to do

  matrix.invert();

Thanks to Concrete Forms for the photograph

and we're done.

Keith knew who he was talking to; I needed the obvious stated. Turns out, communicating with people is what's hard. That's why, instead of putting up walls, SAS is building a foundation for making communication easier.

Last week, I was privileged to be on stage at the Ragan Communications "The Role of Communications in Creating Best Places to Work" summit on the SAS Campus, where I spoke on a panel about blogging at SAS. Participating gelled my thoughts about how our internal blogging community supports our software development and quality initiatives.

Becky Graebe facilitated the panel, and Alison Bolen and Chris Hemedinger provided the gravitas and experience. I was there in case a rogue game of Euchre broke out, or because I've been blogging "djblog: main public static void" internally at SAS since 2005.

Becky highlighted the support that her Internal Communications team, working with our Information Systems Division, provides for us in this space.

  • Anyone is allowed to request a blog at SAS. Simply fill out and submit a web form. Boom, a day later, you're blogging. We have about 500 active bloggers.
  • SAS supports a centrally managed blogging platform (Serendipity).
  • Internal blog content is aggregated into a "SAS Planet" portal; "SAS Moon" aggregates blog comments.
  • Our blogs are indexed by our intranet Google search appliance, making it easy to find content, such as "Nobel Prize winners used Scotch Tape"
  • Shy folks can participate in group blogs until they have built up their confidence.
  • Corporate guidelines create a level playing field and set common expectations. For example, we avoid naming SAS customers by name, etc. Common sense rules. (Hey! that can be scanned two different ways; both work.)
  • Our intranet video portal hosts training videos, webcasts, etc.
  • Alison's job as Editor of Blogs and Social Content is to facilitate blogging; she coaches bloggers. Think about how incredibly useful that is. The fact that we have someone with that role is incredibly telling.
  • Blogs are featured on the "Featured Blogs" section of our intranet home page.

This shows SAS' commitment to open communications. Our internal blogosphere is quite active and one of the places where great ideas are voiced; this often results in improving the way that we develop our software.

 

Stirring the pot

Nascif Abousalh-Neto, a software development manager in Business Intelligence Visualization, recently wrote Steal This Blog for his "Castles in the Air" blog, in which he called for collaboration from our peers. Nascif wrote,


I thought about running a little experiment. What if we could get the expertise of the entire company (OK, the subset that reads this blog) to collaborate in getting to the root causes [of a development process problem]

This post got a lot of attention, and despite it pointing out some flaws in what we do, all the feedback was positive. Paul Kent, VP Platform Research and Development wrote in his executive blog,


Actually, I want you to steal a blog from Nascif called Steal This Blog. He worries that not enough people read his blog. I worry that not enough people will chip in their ideas...

While other comments were along the lines of, "Thanks Nascif for stirring the pot." and "Wonderful topic Nascif!", Nascif noted:

The important comments were not the ones about it being a good idea. They were nice of course, but the really good ones were when people shared their views on the root cause issues that need addressing. ... this was really where the value of the experiment was IMO, to get people to come out and share what they view as opportunities for improvements on their own corner of the big SAS universe.

Indeed.

 

Blog it forward

After our panel, Mark Ragan asked me why I blog at SAS. I noted that SAS has great benefits which many people hear about; those benefits include this nice infrastructure for sharing and collaborating. I feel it is my duty to blog, to voice my opinions, to strive to make SAS (the company and the software) better. I'm very glad that so many others here feel the same. While communication may still be hard, at least joining in here is easy.

As Wayne Gretzky noted, "You miss 100 percent of the shots you never take."

Post a Comment

Keep track of that

I've been a longtime fan of the WNYC radio show, Radio Lab. Hosts Robert Krulwich and Jad Abumrad have interviewed famed biologist E.O. Wilson a few times. Listening to him describe his study of ants and how they make tracks and trails is a great story of perseverance and attention to detail. E.O. Wilson had a bit of a head start on us, but here at SAS, we're pretty good at tracking bugs, too.

No one really likes to admit it, but even SAS has defects in our software, although we do our best to find them all during development. So, when a developer or a tester (or even the occasional manager, pointy-haired or not) finds a bug, they enter it into our bug tracking system. Once there, we, well, track them. This is the story of our bug tracker, DEFECTS.

Choosing a new defect's component. Click the image to see a larger screenshot.

The image to the right shows a small region of our DEFECTS web client that we use for entering new defects. Here, I'm selecting the defect's component JAVA.PROCESS from several alternatives which match what I've typed so far, the input "JAVA.P". The system uses Ajax to provide this incremental completion. The blue asterisks mark required (and predictable) defect description fields. The defect entry web client is implemented using Google Web Toolkit. Click the image for a larger screenshot showing more of the current user interface. In a future entry, I'll discuss the evolution of DEFECTS to show how it came to its current form. (DEFECTS has evolved faster than any species of Formicidae.)

The second half of the defects system is the reporting side. We have a number of clients which provide varying degrees of power for tracking defects. The broadest (imagine a young E.O. Wilson looking at ants with his first magnifying glass) allows users to just perform plain text searches of a repository of HTML-formatted defects via a Google appliance. Moving up to microscopes, users can search for defects by entering product names or specific lower-level component names. The scanning electron microscope tracker allows users to enter arbitrary SQL where clauses or SQL queries to find very precisely specified sets of defects.

Not your average spay/neuter program

Once we have tracked the bugs, the next step is to fix them. No, this is not like fixing a stray dog; I mean repairing the underlying software faults.

The DEFECTS tool plays a crucial role in this process. All code for SAS products is stored in our source management systems; we have several CVS servers for this purpose. Developers implement fixes or new features on their workstations, then test them. When they are confident their changes are correct, they return to the DEFECTS system and change the defect's status to a REQUEST. This sends a signal (email) to the testing contacts associated with the software component and defect. The quality assurance team and management reviews the request and if they are satisfied that sufficient tests have been run and that the change will not have an adverse effect elsewhere, they change the status to APPROVE. The defect's unique identifier now may be used as a FIXID—an identifier for the fix. In order to commit code changes, our CVS systems validate the commit requests. All requests require a comment that matches the checkin request: the comment must begin with "FIXID <FIXID>:". If this is missing or the defect has not been APPROVE'd, CVS rejects the commit. Once committed, the developer and tester work to validate the fix against the full system software build, and when done, mark the defect as FIXED.

Finally, we can query our system and list all the source files that have been changed using a FIXID—a very useful code auditing feature. (Sometimes, this feels like an overly heavyweight process. In the future, I'll discuss Agile software development at SAS.)

A critical tool

A R&D director once told me that introducing new tools into SAS R&D is hard because so much of our workflow and information management is focused on two key tools: email and DEFECTS. In practice, DEFECTS is used for tracking more than bugs. All new features for our products go into the DEFECTS system, as do redesign and refactoring work to make the code perform better or to render it more maintainable or reusable. DEFECTS is used to track change requests that have potentially wide impact and the related work to support development of SAS' myriad products—such as creating new branches in our source management system, requesting new installers, or adding new third party binaries in support of our Java and Adobe Flex-based product development.

Almost all changes to our extensive code base and our development tools and processes, can and are tracked with DEFECTS. That's a lot of work being done by the 2,000 or so people doing product development, testing, installation, and deployment. Next, I'll explore this aspect of tracking bugs.

By the numbers

DEFECTS is powered by SAS
My colleague Chris Hemedinger (The SAS Dummy) mentioned how we
eat our own dogfood”—that is, the DEFECTS system, primarily written with SAS, is a great vehicle for validating the SAS platform.

Here's some information from Dick Wiersma, the development manager. On a typical weekday:

  • the web server handles about 850,000 HTTP requests.
  • the SAS/SHARE server writes more than 7,000,000 lines to the log file (with verbose logging enabled).
  • SAS/SHARE processes about 375,000 SQL queries from SAS/IntrNet htmSQL web pages. These are for various reports. We have a library of stored reports for listing defects by product, component, developer, department, or any of a wide number of other categorization fields.
  • the server handles about 150,000 separate connections from client applications.

To meet this demand, the DEFECTS system's SAS/SHARE server uses SASFILE statements to load the entire database—about 26 gigabytes— into memory. SAS is run using a MEMSIZE value of 40GB. The tables in the DEFECTS database vary from a handful of records to 55 million records. Dick shared some of his team's experience tuning the SAS/SHARE servers in Performance tuning for SAS/SHARE servers.

The Open/Closed Principle

Software developers may be aware of the Open/Closed Principle, first defined by Bertrand Meyer and later refined by Robert C. Martin. This useful design principle states that software systems should be open for extension or reimplementation, but closed for modification.

An inversion of this principle applies to our DEFECTS system, to great benefit to us, but also with some costs. The big benefit of our defects system is that it is open: the set of SAS data sets which contain the defects data, their schema, and even the server where the data is stored, is open. For example, there are ten key tables which contain the essential data about stored defects, plus several auxiliary tables which serve as lookup tables for input validation. For example, one table
lists the components including JAVA.PROCESS (see the defect entry screen shown above). We can write new tools to query the data and generate reports. This has created an ecosystem of productivity tools: managers can easily find and track defects that they care about. As an example, we have a web tool that lets users create ad-hoc reports (using SAS htmSQL) by choosing key fields to query on (such as product, department, or employee) and users can save those queries for later execution.

Of course, this degree of openness comes at a cost. It is difficult to change the schema because doing so could break tools and reports that so many depend on. For example, one of the key fields in the defects data set is the text field which holds the name of the software component. Since DEFECTS was designed when the SAS system only supported eight character variable names, this variable has the name "CMPONENT" instead of "COMPONENT", and we must live with this awkwardness today, even though the SAS data set format has supported longer variable names for many releases. However, this is a cost we are willing to take because our system is an internal one, and the benefits of the openness outweigh the benefits of higher level abstraction that would otherwise hide the schema behind an abstraction layer.

Designing such a robust, complete, and consistent framework and API is difficult and intricate work, with many benefits, but sometimes we don't have the luxury of designing that way, and forcing all users to use such an API might just make the system harder to work with and less open.

This openness also partially explains why we do not use a commercial or open source software product like Bugzilla, Trac, or JIRA. While these are fine tools, switching to these products would be very disruptive—they would be incompatible with so many of our existing tools and reports, and they would not be as open and as extensible. And, they would be a different brand of dogfood, so to speak, and we know it's hard to teach an old dog to eat new dog food: it's a classic case of vendor lock in.

Trail's end

Tracking defects is essential to SAS' quality imperative, and it is an integral part of every developer and tester's experience. I hope this revue has given you a sense of what bug tracking at SAS is like. I think E.O. Wilson might be proud.

Watch the Peer Revue playbill for an upcoming discourse on the evolution of DEFECTS.

Post a Comment

Peer Revue

Massage Therapy at SAS

Perhaps you are like me and have gotten top-ten list fatigue from the typical press that SAS gets. The pervasive coverage leaves the impression that we all arrive on campus at 10:00AM, grab a massage before heading to the natatorium for a few laps, take a stroll around the lake before going to the cafeteria for a subsidized lunch while listening to a piano recital, spend some time at the book exchange, then head home at 1:30.

Apparently, the only ones who do any real work around here are the pianist and the massage therapists... and those writing the press releases.

I present Peer Revue to disabuse you of that myth.

Currently in SAS' Advanced Computing Lab, I have worked in research and development since joining SAS in 1989 (which sometimes makes me a new guy on the block). What excites me about SAS is not the benefits and culture, as much as I truly appreciate them. Rather, it is the technical challenges and really hard customer problems we work on, and the cool environment that helps up create the great software that ultimately pays for those benefits. My particular passions are in the areas of programming languages, concurrency, software productivity and quality tools, and collaboration.

Peer Revue will contain a good deal of actual peer review, where I will recount how my peers get the work of SAS research&development done and reveal what tools, techniques and technologies we use. SAS has a unique history, and I'll use this space to recount some of that history and how it has shaped our software development practice and tools. It's a story I think will interest anyone curious about how the world's largest privately held software company works. I'll even throw out leads on interesting papers my peers have submitted to peer-reviewed journals.

Peer Revue will have the feel of a revue—similar to live stage revue, but with less polish—which will be enlightening and entertaining. Consider me your master of ceremonies.

For example, look for revues on

  • how we'd make E. O. Wilson proud,
  • why DDT is not for pest control at SAS,
  • Spirits in the Virtual World, a variation on an old Police song, and
  • the Demise of the Gansy of Gnomes Gardening Club
Old Curmegeon beer

Before going further, I must disclose that I have a reputation at SAS for being a bit of a curmudgeon, not unlike ACM Queue's Stan Kelly-Bootle and George Neville-Neil, only less likable and eloquent. (Perhaps I should hyphenate my name, David Bie-Sack, for optimal curmudgeon transference.)

Inside SAS' walls, I'm not shy about pointing out where I think our tools and processes need improvement. In Peer Revue, I'll try to present the same honest depiction of what we do in SAS research and development. I'll discuss what works well and what doesn't.

Krispy Kreme donuts get a coating of sugar
Krispy Kreme's famous sugar coating process surpasses that of Peer Revue.

Image by William Lawrence

However, I respect and like my peers, so I won't be too harsh. (When I sugar-coat a problem, I'll flag it with the Krispy Kreme Alert.)

This honesty is entirely selfish: my hope is that my readers will chime in and either confirm that these are unsolved problems, or more likely, point out some useful ways they have innovated in this ever-changing milieu of building complex software systems.

Don't get too excited, though. If you tune in hoping to see someone in this revue fall off a thin tight rope from tremendous heights, seek your thrills elsewhere. I won't be revealing SAS trade secrets and putting my personal employee benefits (my salary being the most important one) at risk. Naturally, the opinions in Peer Revue are my own, not those of my employer.

Please leave a comment below if there is something you would like to hear about or if you have comments to share, or contact me if you prefer a less public forum.

With that prologue, let the revue begin.

Post a Comment