Five data quality archetypes, part 2
@philsimon on the need to recognize DQ differences.
English
Hidden in Plain Sight
By now most of us are at least aware of the fact that too much sugar isn’t good for us. But you may not be aware of some of the hidden sources of sugar like "not so sweet" foods such as pasta sauce, salad dressing, ketchup and bread. Some other
Five tips from Simulating Data with SAS
Data simulation is a fundamental technique in statistical programming and research. My book Simulating Data with SAS is an accessible how-to book that describes the most useful algorithms and the best programming techniques for efficient data simulation in SAS. Here are five lessons you can learn by reading it: Learn strategies
Forming data quality habits in 2015
It’s common at the start of a new year to create a long list of resolutions that we hope to achieve. The reality, of course, is by February those resolutions will likely be a distant memory. The key to making any resolution stick is to start small. Create one small
Panning for corporate gold requires gold standard skills
We now live in the era of ‘big data’, where data and its analysis have become crucial to the modern economy. In fact, "big data is the new 'corporate gold'," according to Mark Wilkinson, managing director of SAS UK & Ireland. A recent study by Cebr found that companies in
Finding matrix elements that satisfy a logical expression
A common task in SAS/IML programming is finding elements of a SAS/IML matrix that satisfy a logical expression. For example, you might need to know which matrix elements are missing, are negative, or are divisible by 2. In the DATA step, you can use the WHERE clause to subset data.
2014 media coverage explores who uses SAS and how
It’s that time of year again when we look back and consider how accurately and extensively the SAS story was covered in the media over the past year. It’s not always a simple or predictable story, but it is usually interesting. Consider some of these threads: How did companies or
Millennials will outnumber Baby Boomers in 2015
To get into the mood for this blog post, you should first listen to the music video of The Who singing My Generation... I guess everybody has 'their generation' and here in the U.S. the most famous generation has been the Baby Boomers. Many companies have tried to design products they
Displaying Unicode Symbols in Legend
Including special Unicode symbols into the graph is getting more popular. In general, SG procedures support Unicode strings in places where these strings are coded into the syntax such as TITLE, FOOTNOTE. These support Unicode characters and also the special {SUP} and {SUB} commands. This is because these statements are rendered
Jedi SAS Tricks: DS2 & APIs - GET the data you are looking for
While perusing the SAS 9.4 DS2 documentation, I ran across the section on the HTTP package. This intrigued me because, as DS2 has no text file handling statements I assumed all hope of leveraging Internet-based APIs was lost. But even a Jedi is wrong now and then! And what better
FVA Interview with Jonathon Karelse
In December the Institute of Business Forecasting published the first of a new blog series on Forecast Value Added. Each month I will be interviewing an industry forecasting practitioner (or consultant/vendor) about their use of FVA analysis. The December interview featured Jonathon Karelse, co-founder of NorthFind Partners. Among his key
Cloud: 4 deployment models
This is the last of my series of posts on the NIST definition of cloud computing. As you can see from this Wikipedia definition, calling anything a “cloud” is likely to be the fuzziest way of describing it. In meteorology, a cloud is a visible mass of liquid droplets or frozen
Learn how to maximize your data with SAS and Hadoop
Outside, the Cary, NC sky is gray and winds are blowing freezing rain, but a group of statisticians at SAS are channeling warm green hills and the soft, gold light of a California evening. Team conversations alternate between distributed processing, PROC IMSTAT and how many pairs of shorts to pack.
Static Models and Dynamic Data
After working in the data quality industry for a number of years, I have realized that most practitioners tend to have a rather rigid perception of the assertions about the quality of data. Either a data set conforms to the set of data quality criteria and is deemed to be acceptable
Using SAS analytics to monitor blog posts
As a blogger, I often wonder whether my blog posts are 'successful' - and being a graph guy, I like to visually analyze the data, to try to answer that question. The most common measure of a blog post is probably the number of times it was viewed, so I guess
Big data preparation, big data quality and big data governance, oh my!
This isn't Kansas anymore. Oz has become a sprawling, smart metropolis filled with sensor data. How do we make sense of, clean, govern and glean value from this big data so we can get Dorothy home? The answer is SAS Data Management. With the latest portfolio updates, customers will be
Getting personal at the touch of a button
Over the past decade in insurance, the rise of the aggregators (organisations that compare quotes between different insurers) and direct channels has had a profound impact on personal lines distribution in the UK. However, personal lines brokers remain a critical route to market, especially at a time when many insurers
Wanna solve the US budget deficit? Fix fraud!
Here’s a great way to kill a conversation at the next cocktail party you attend. Start talking about the US budget deficit. You remember the deficit, right? It’s the difference between what the government collects and what it spends. In 2014, the US federal government spent $488 billion more than
Don't let difficult graph legends get your goat
Is this blog post about techniques to use on difficult graph legends, or is it about goats? The answer is both! But first, to get you into the proper mood, here is a picture my friend, Mark, took of some cute goats. And some links to YouTube videos about goats
Marker Symbols
There has been much discussion on the SAS Communities page on usage of different symbols in a graph. The solutioin can vary based on the SAS release. New features have been added at SAS 9.4 releases to SG Procedures and GTL that make this very easy. With SAS 9.4M1, almost any combination is
SAS Grid Manager, shared-nothing storage and Hadoop: Part 1
When designing a SAS Grid Manager architecture, there is a requirement that has always been a critical component: a clustered file system. Over the years, vendors have released versions of these systems that are more robust and SAS has increased the minimum IO requirements, but the basic design has never changed—until now.
Soft Drinks, Diet Sodas and Juices, Oh My!
Healthier Alternatives to Quench Your Thirst With research linking sugar to heart disease, diabetes and GI imbalances, many of us are trying to reduce or eliminate sugar. And to add salt (or sugar) to the wound, new studies are suggesting that some artificial sweeteners ALSO have negative health consequences. When
Calling a global statement inside a loop
The other day I was creating some histograms inside a loop in PROC IML. It was difficult for me to determine which histogram was associated with which value of the looping variable. "No problem," I said. "I'll just use a TITLE statement inside the loop so that each histogram has
Solving Sudoku with SAS/IML – Part 1
Sudoku solvers have been written in SAS using a variety of methods (e.g., the DATA step, PROC SQL, and PROC CLP). Surprisingly, SAS/IML appears to have been overlooked for this purpose. On a challenge from a coworker, I wrote this blog post to demonstrate the flexibility of SAS/IML in the
Crowdsourcing data improvement: Part 1
James Surowiecki wrote a book about The Wisdom of Crowds. Jeff Howe, who co-coined the term crowdsourcing, wrote a book about Why the Power of the Crowd Is Driving the Future of Business. In this blog series, I explore if it’s wise to crowdsource data improvement, and if the power of the crowd can
Diagnosis: Your data is not “normal”
“Let’s assume a normal distribution …” Ugh! That was your first mistake. Why do we make this assumption? It can’t be because we want to be able to mentally compute standard deviations, because we can’t and don’t it that way in practice. No, we assume a normal distribution to simplify
Using SAS to help locate the 100 best restaurants in the US
I recently read an article that listed the 100 best restaurants in the US - but the article didn't have a map. I decided to use my SAS skills to change that! When it comes to restaurants, I eat out a lot (and by 'a lot' I mean I never
Twelve posts from 2014 that deserve a second look
I began 2015 by compiling a list of popular articles from my blog in 2014. Although this "People's Choice" list contains many interesting articles, some of my favorites did not make the list. Today I present the "Editor's Choice" list of articles that deserve a second look. I've highlighted one
Five data quality archetypes: Part 1
.@philsimon on the different folks you'll encounter in many large organizations.
Internet of Things: A game changer for insurance
~ This article is co-authored by Binod Jha, Global Product Manager for Insurance Solutions at SAS, and Amol Kokane, Senior Development Manager for Insurance & Risk Management Solutions at SAS ~ How might insurance policies change if sensor data could be automatically transmitted and analyzed from your car, your home and even your