Considerations and tips for big data integration
.@philsimon asks some fundamental questions about taking the next step with #bigdata.
Uncategorized
Machine Learning: An invited guest to the IoT party?
Research indicates that IoT and Machine Learning are more valuable to utilities when used in combination but there are hurdles to overcome first. Machine learning and IoT will enable utilities to better realize the next generation of the power grid: a distributed system with power flows among millions of things
How social brokers is changing insurance
“All for one and one for all” is best known as the motto from “The Three Musketeers”, but this phrase could easily sum up the growing trend in social brokers. With advanced analytical techniques like generalized linear modeling insurance companies have created more granular pricing structures. But despite the assertions
Ten tips before you run an optimization
Optimization is a primary tool of computational statistics. SAS/IML software provides a suite of nonlinear optimizers that makes it easy to find an optimum for a user-defined objective function. You can perform unconstrained optimization, or define linear or nonlinear constraints for constrained optimization. Over the years I have seen many
Jedi SAS Tricks: DIY Tasks in SAS Studio
In my previous post, Jedi SAS Tricks - Make This a Button in Base SAS I demonstrated running a SAS program from a tool bar button in the SAS Windowing environment. The program we execute is the macro from a previous post, Jedi SAS Tricks: The DATA to DATA Step
Back to the future: From CCAR to enterprise risk management
As everyone in the financial industry knows, CCAR (comprehensive capital analysis and review) was set up as a capital adequacy assessment for the major US banks. Through several years of practice and feedback, CCAR has evolved into a comprehensive stress testing exercise -- and in doing so, created a new
Where should data quality happen?
In my previous post I discussed the practice of putting data quality processes as close to data sources as possible. Historically this meant data quality happened during data integration in preparation for loading quality data into an enterprise data warehouse (EDW) or a master data management (MDM) hub. Nowadays, however, there’s a lot of
What is a DATA step view and why is it important?
Last week I analyzed 12 million records of taxi cab transactions in New York City. As part of that analysis, I used a DATA step view to create a new variable, which was the ratio of the tip amount to the fare amount. A novice SAS programmer told me that
What comes after Y? Getting to know Generation Z
The current rate of workforce change is unprecedented. Each new generation of working professionals is emerging more rapidly than the one before, with increasingly advanced expectations of their employers. The latest cohort is Generation Z, typically defined as those born after 1998. This group is fresh-faced, independent and extremely pragmatic.
Education shines BIG in Vegas!
Not everything that happens in Vegas stays in Vegas. At this year’s SAS Global Forum 2016, so many exciting things happened that I have to share some highlights. From the use of analytics in college administration to the teaching of analytics in the classroom, the range of information presented was quite
IoT: How could your company benefit from real data accuracy?
We had just completed a four-week data quality assessment of an inside plant installation. It wasn't looking good. There were huge gaps in the data, particularly when we cross-referenced systems together. In theory, each system was meant to hold identical information of the plant equipment. But when we consolidated the
One more ‘giant leap’: Four decades of cardiovascular research data now shareable
In 1969, a human walked on the moon. I remember my parents waking me in the wee hours to watch (albeit bleary eyed) the grainy images on our black and white television as Neil Armstrong set foot on the lunar surface. Heady stuff then and now. In another world-altering advancement,
May the 4th be with you
Most folks who know me, know I'm a bit of a Star Wars geek. I've analyzed the original trilogy scripts and documented my findings in a paper called Star Wars and the Art of Data Science. I'm always looking for excuses to get my hands into Star Wars data, and
Create a package in SAS/IML
In a previous post I showed how to download, install, and use packages in SAS/IML 14.1. SAS/IML packages incorporate source files, documentation, data sets, and sample programs into a ZIP file. The PACKAGE statement enables you to install, uninstall, and manage packages. You can load functions and data into your
Could a recommendation engine pick 'The Cuckoo’s Calling' as a best-seller?
How many of you have read The Cuckoo’s Calling by the previously unknown author Robert Galbraith? The answer is not many, until it came out that Robert Galbraith was none other than blockbuster best-selling author JK Rowling. Sales then skyrocketed. Rowling recently published some of the rejection letters she received as
How will the IoT affect your data management strategy?
.@philsimon says that it's never too early to think about the IoT and data management.
Three ways to monetise your data
There’s no such thing as a free app. “What?” I hear you say, “but I download free apps all the time!” So then why do organisations spend considerable time and effort creating free apps? Often their goal is to collect data and turn it into money. Consider this example. There’s
SAS Author Spotlight: Goutam Chakraborty
As the excitement of SAS Global Forum begins to die down and we dust off our sequins for another year, it’s time to get back behind the desk. This year at SAS Global Forum we hosted a "Top Tips from Your Favorite SAS Press Authors" lunch where we asked three or four
Does big data spell big trouble on the campaign trail?
The timeline on the latest season of Netflix’s series House of Cards has finally caught up with the real world, and the current plot line regarding President Frank Underwood’s underhanded dealings to win the Democratic nomination has many parallels with the current US primary election coverage saturating TV and print
How much do New Yorkers tip taxi drivers?
When I read Robert Allison's article about the cost of a taxi ride in New York City, I was struck by the scatter plot (shown at right; click to enlarge) that plots the tip amount against the total bill for 12 million taxi rides. The graph clearly reveals diagonal and
Jedi SAS Tricks - Make This a Button in Base SAS
A recent post, Jedi SAS Tricks: The DATA to DATA Step Macro, engendered a lot of response on Twitter. One of the re-tweets included a call to action - make this a button in Base SAS!
All quiet on the Barnett Front
The Barnett Shale in North Texas hit a historic mark on April 25: Its rig count fell to zero. Two hundred rigs once harvested the 40 trillion cubic feet of natural gas in this massive basin, stretching beneath 17 Texas counties. Today, nothing. This dramatic silence in North America’s second-largest
What does PROC REPORT do?
At SAS Global Forum in Las Vegas I was asked the question, “What does PROC REPORT do?” It is a simple question, but I hesitated to answer. I’m normally so deep inside the nitty gritty details of PROC REPORT that I don’t often think about what it would be like
Analytics in the news: the NFL draft
As American football teams prepare to select new team members later today, fans and pundits can only guess how the draft will turn out. Will your favorite professional team make good picks? And will your favorite college players go to good teams? With high stakes and billions of possible outcomes,
Filtering the signal from the noise: The lifesaving value of experience when analytics matter
The expletive leapt from the lips of the orthopedic surgeon as he entered my hospital room. He was swathed in a green gown, purple gloves, and a white surgical mask – for his protection, not mine. He was clearly upset. “What, exactly, does $#@! mean?” I asked. “It means you’re
Could text analytics be the cutting-edge technology the oil and gas industry was waiting for?
Small causes can have large effects; or how a discovery in the Barnett Shale can spike some interest in the rest of the world and change the face of the industry. This article is co-written by Sylvie Jacquet-Faucillon, Senior Analytics Presales Consultant, SAS France; and David Dozoul, Senior Adviser
How to partner with IT to build a dashboard for community college administrators
It's a common problem in any industry: getting a large number of similar requests for information. But with limited resources and an already overburdened staff, how do you handle it? At El Paso Community College, analysts from the Institutional Research (IR) team enlisted the help of IT to create a data
Characteristics of IoT data quality
In my last post we started to look at two different Internet of Things (IoT) paradigms. The first only involved streaming automatically generated data from machines (such as sensor data). The second combined human-generated and machine-generated data, such as social media updates that are automatically augmented with geo-tag data by
Taking the Man out of Daten-man-agement
These: Zum Daten-MAN-gen braucht man Software. Hat man mehr Daten, nimmt man mehr Software… „Mehr vom selben!“ – jener Kulturtechnik aus der Ecke „viel hilft viel“. Stimmt das so? Man kann dies‘ knifflige Thema von verschiedenen Seiten beleuchten – im Folgenden einige mehr oder minder ernste Ansätze dazu:
Packages: A new way to share SAS/IML programs
My previous post highlighted presentations at SAS Global Forum 2016 that heavily used SAS/IML software. Several of the authors clearly want to share their work with the wider SAS analytical community. They include their SAS/IML program in an appendix or mention a web site or email address from which the