Event stream processing: a life raft in the data deluge

Niagra Falls, Ontario, CanadaDo you ever have stress dreams? You know, where you’re taking an exam for which you haven’t studied, or you’re forced to wait tables in a sea of angry restaurant customers?

For many of us, the stress nightmare of the modern era involves trying to make sense of a never-ending stream of data. Being asked to make decisions based on a constant flow of ever-changing variables – that’s not only stressful, it’s the downside of the information barrage that is the Internet of Things. Getting it right is tricky – and worse than just being unprepared. It’s like trying to navigate a boat at the edge of Niagara Falls.

That’s an apt analogy because more than 3,000 tons of water crash over those falls every second. The volume of streaming data is even bigger. With the Internet of Things pushing out data from more and more devices, streaming data is transmitted continuously at rates of millions of events per second. Storing this data and then analyzing it later is just not practical in this new world.

Big data, big challenge

When it comes to crunching all that data, and doing it fast, the challenge is immense. The brilliant people in SAS R&D saw what was coming and knew what was needed: software that processes and analyzes data in milliseconds or even microseconds – before it is stored. We call it SAS® Event Stream Processing. To date, this solution has helped many of our customers remain competitive and maximize the value of their goods and services.

One example I like to share is a consumer bank’s fight against fraud. The bank is using SAS Event Stream Processing to detect, monitor and address suspicious transactions and fraudulent behavior in 8 million online banking accounts. With 1 million payments per day, the bank sees an average of 35 transactions per second, with a peak of 230 transactions per second. Thanks to real-time alerting, these suspicious transactions are put into an investigation queue and addressed within 30 minutes. As I’ve noted before, fighting fraud is something that has to happen in the moment. Catch it after the fact, and you’re way too late.

And in some cases, midstream processing is actually saving the world. None of us want to see another offshore drilling accident, right? That’s why a leading energy company is using SAS Event Stream Processing to continuously monitor the performance of its electrical pumps in offshore oil platforms. More than 2 million sensors on hundreds of platforms yield about 3 trillion rows of data per minute. Crunching all that data, the SAS solution was able to predict – and thus prevent – a pump failure, protecting the ocean and saving the customer millions of dollars in the process. I’m proud of that.

Cleaning while streaming

So how do we actually do it? Half the battle is cleaning the data as it’s streaming. Sensors can give false readings, and data can be inconsistently formatted. Normalizing data while it’s flowing is a key part of success because that’s when you’re able to detect patterns and define priorities. SAS Event Stream Processing determines what data is relevant, if and when it needs immediate action, where it should be surfaced for situational monitoring, and where it should be stored for more in-depth analysis. And it all happens in the blink of an eye.

Pretty cool stuff.

The reason I’m so pumped about event stream processing is that the stream is only going to get bigger. Analyst firm Gartner Inc. predicts that the Internet of Things will grow to 26 billion units by 2020 – an almost 30-fold increase from 2009. Not sure what that really means? Consider it this way: We’re already drinking from a fire hose. Soon we’ll be trying to sip from Niagara Falls.

I, for one, am glad we’ve got help.

More on event stream processing

For more information about how SAS can turn streaming data into actionable insight, check out the following resources:

Post a Comment

5 questions to prepare you for the Internet of Things

42-45884927In my last post, I talked about the Internet of Things, and how the data streaming from cars, planes and industrial machines can be used to improve business decisions.

With the number of streaming data sources growing by 20 percent annually, I’m not the first person to notice this trend. As executives in every industry start to see the potential in these data streams, they’re all asking the same types of questions.

Recently, I’ve talked to banking CEOs, telecom CIOs, retail CMOs and heads of government agencies, and the conversations are very similar. Primarily, they’re asking, "How can I use all this data to discover new opportunities?" More specifically, the questions – and basic answers – sound like this:

Q. With all the data out there, how can I store it efficiently?
A. Hadoop.

Q. What if I need the data right away? How can I get it quicker?
A. Streaming data.

Q. Now that I have access to all this data, where do I start?
A. Data visualization.

Q. How can I use this data to discover new possibilities?
A. Advanced analytics.

Q. How can I get my analysis done quicker to get a jump on the competition?
A. In-memory, distributed processing.

Let’s look at each of those answers a bit more closely:

Hadoop. I cannot over emphasize the importance of understanding what you’re capable of doing with Hadoop. The best way to work with Hadoop is to create an analytical platform, so you can do more than just store your data there. You need to be able to access data in Hadoop, run analytics inside the Hadoop environment, inside the cluster, and even run in-memory calculations inside the Hadoop cluster.

Data management. Data management for analytics is not the same thing as data management for an enterprise data warehouse. Analytical data management adds value along the way by completing summarizations and adding metadata to variables before putting them into memory.

Visualization. Visual analytics provides capabilities beyond general business reporting, by giving you a way to explore and understand all your data. Visual statistics takes it even further by making it easy to explore, discover and predict by implementing statistical algorithms without having to write any code.

Advanced analytics. This one is a no brainer. To see real value in your data, you need to move beyond basic analytics to optimization, forecasting, text analytics, event stream processing and more. With a wide range of advanced analytics at your disposal, you’ll reveal optimal and lucrative opportunities, expose risks, deepen customer understanding, and deliver predictive insights.

In-memory, distributed processing. Now that Hadoop is easily available, storing your data is no longer an issue. The real issue is whether you can process it quickly enough. In-memory processing can help you keep up with increasing data demands. You can not only do things quicker but you can do new things -- and change the way you’ve always done things in the past, so true innovation can happen.

Your competitors might be looking at one or two of these areas, but if you can develop a strategy that excels in all five, you’ll operate more efficiently, make faster, smarter decisions, and get big value out of data from the Internet of Things.

Post a Comment

The Internet of Things is not just for consumers

Close up of clothes dryer buttons

Will your dryer be connected to the Internet?

One of the buzzwords we continue to hear a lot about is the Internet of Things. Most of us have heard the term by now, and we’ve read about the consumer benefits of smart thermostats or smart cars that transmit and receive data, and make adjustments automatically. But what are the real benefits here? Is this really just a story about changing the way I heat my home?

Actually, there’s a tremendous amount of data being generated from the Internet of Things, and smart businesses are just starting to realize the potential. At the Consumer Electronics Show in Las Vegas last month, the CEO of Samsung said that 90 percent of what they produce will be connected to the Internet by 2017. It’s not just TVs; it’s everything: home theaters, washers, dryers. Everything that they do. And another couple of years after that, 100 percent of what they produce will be connected to the Internet.

It must be serious business. Because Samsung’s competitors are already being accused of sabotaging their washers in stores. Wow. It is getting tough out there with these devices.

It’s funny, but at the same time, it’s for real. If you don’t think the Internet of Things is for real, think about what Apple has done with Apple Pay, Apple Healthkit and Apple Homekit. People have wondered how Apple will continue to succeed with phone sales alone.  It’s not about the phone. It’s about the ecosystem, and changing the way people do business. Analytics will be a large part of that ecosystem.

The Internet of Things is not just about futuristic and superfluous features on consumer devices. This is something we should all be taking advantage of, and it goes well beyond the realm of the consumer.

You could look at every industry and pick ten examples, but I’ll name just a few.

  • In the oil and gas industry, a modern drilling platform generates 8 terabytes of data a day. They’re obviously monitoring this data in some control room, but what about applying analytics to that data stream? Now you can understand what’s about to happen and predict failures and save a whole lot of money.
  • In the airline industry, the Boeing 787 generates 40 terabytes of data an hour. Every single component is connected to the internal backbone or the network within that airplane. Why is that important? Predictive maintenance is just one example. Airlines can identify potential failures before they occur and improve air safety for everyone.
  • In the automotive industry, connected cars are generating a gigabyte of data a second. What are we going to do with that? There are all sorts of possibilities. Everything from insurance companies understanding driver habits to the part suppliers understanding how the components are performing can be improved. Plus, the manufacturer can use streaming data to work with the consumer to keep everything working optimally from a maintenance perspective.

These data streams are going to change every industry. And “stream” is the operative word. This is not data that you put into a data warehouse to store for future analyses. This is data that you analyze as it flows into the network so you can feed automated decisions back to the devices and to the business.

The Internet of Things will bring streaming analytics and event stream processing into the mainstream. As businesses look at the opportunities in the Internet of Things, they will find that the real advantage comes in the “Analytics of Things.”

Where are you seeing the Internet of Things in your industry? What ideas do you have for analyzing streaming data? You can get more ideas by reading this article about the Internet of Things. Or download an in-depth report about the use of sensor data in multiple industries.

Post a Comment

Q1 2015 Intelligence Quarterly: Doing good with data

Q1 2015 Intelligence Quarterly

There is widespread evidence that big data analytics helps achieve short- and long-term development goals around the world. Poverty, disease, hunger, illiteracy and many other global development challenges all benefit when analytics are applied.

In this blog post, I’ll address not only how to use data for good causes, but also how to protect public data and avoid any misuse or unintended negative uses of personal data. Keeping data safe and secure is a valid concern that cannot be ignored, and we need to address both the good and the bad in our plans for using data to improve the human condition.

What is ‘data for development’?

Data for development is a growing initiative that applies big data analytics to improve policies and infrastructures for health, welfare, safety and security all around the world. For example, can we make cyberspace safe through monitoring and governance? Could we reduce the cost of health care while improving the quality of care and saving lives? Can we make the world safer for our children and especially at-risk youth in our communities?

In the Q1 issue of Intelligence Quarterly magazine, we draw our attention to a subset of the data for development movement and look at the ways big data analytics is used to improve safety and security, including:

  • Preventing cyberattacks.
  • Combating financial crime.
  • Thwarting terror threats.

For these programs to work, public and private organizations need access to data and analytics solutions that can offer greater speed, frequency, detail, accuracy and information sharing.

As these tools become more readily available, we also must mitigate the potential misuse of data. This means balancing the individual’s right to privacy with society’s right to security and safety.

‘Data for good’ must prevent ‘data for bad’

Despite the tremendous opportunity for good, there’s significant potential for data abuse and misuse. There is a widespread lack of trust among governments, citizens, international organizations and the private sector when it comes to collecting and analyzing data. These anxieties stem from many sources, including power asymmetries, a perceived decline of privacy, and the potential for misusing data to violate civil, commercial or human rights.

Establishing meaningful and enforceable protections for collecting, storing, processing and sharing data is essential. But how can we work through the complexity and identify key points of focus for progress? Governments, the private sector and the development community can take action by mitigating concerns in three priority areas:

  • Technology. As billions of sensors come online and collect data, and as we analyze and synthesize more of this data, it is essential to build systems with privacy in mind. This starts with an understanding of how data is generated and how it is often most secure at its source. If we can move the analytics to the data instead of moving the data around to be analyzed in multiple, unsecure places, we can help ensure the security and reliability of the data. Essentially, the data doesn’t need to move; the models do.
  • Governance. International norms should be developed to oversee the quality of data, the open sharing of certain types of data and the protection of data to ensure they aren’t altered by political influence. Monitoring and accountability mechanisms should work effectively with broader data ecosystems, through linked interoperability standards, data sharing protocols and requirements for the data at multiple levels of official use and status. Most importantly, these new governance systems will have to work to earn and maintain the trust of individuals and organizations around the globe.
  • Legislation. Clear, robust policy and legal frameworks also must be developed to prevent the misuse of data. One element requiring continual attention is the degree to which data that has been shared is “anonymized.” Somewhat on the opposite end of the spectrum is the right to be counted. Additional legislative actions might include rights for: identity, privacy, data ownership, due process, participation, nondiscrimination, equality and consent principles.

These three areas are emerging as important starting points for large-scale uses of data for good causes. Balancing trade-offs between the public good and potential harm to individuals is hard work, but it can be done. It all starts with communicating the benefits of using data – combined with technology, governance and legislation – to improve safety, security and development efforts around the world.

Read Intelligence Quarterly to learn more.

Post a Comment

The best gift for retailers this season: analytics

Retail analytics Gilt retailer customer intelligence giftIt’s that time of year again, and I have retail on my mind. Not only because we’re reaching the peak of the holiday buying season, but because we at SAS are getting ready for one the biggest events on our calendar, the National Retail Federation tradeshow in January.

As I’m thinking about retail, I’m thinking about the unique ability of analytics to help drive sales. In some respects, brick-and-mortar retailers have an innate advantage over online stores. They can get to know customers face-to-face and understand sizes, tastes and needs on a personal level. To match that level of knowledge, online retailers must turn to the power of analytics.

One of the fastest-growing retailers in the online world is doing just that. Gilt Groupe is a purveyor of luxury goods not typically found online, such as designer samples and overstocks from 2,000 partner brands. You can find everything from high-end gifts and elegant stocking stuffers to one-of-a-kind luxury experiences. Recently I even saw an offer for a guys’ weekend in the Dominican Republic. Hoping Santa brings me that one.

The products are offered in small quantities and at a steep discount, but only to an exclusive audience and only for a limited time.

As a result, things have to be precise, so Gilt is using SAS® Analytics to better understand what its customers want. Decisions need to happen fast because, at Gilt, a new sale starts every day and is quickly over. The company needs to intimately understand which customers are drawn to the site in order to understand what merchandise will be the most appealing. Fortunately, the data was there. Between demographic data, browsing and shopping history, mobile and transaction data, and marketing history, Gilt had what it needed to get started. But data alone doesn’t equal customer insight.

CEOs and CIOs often learn this lesson the hard way, but Gilt had an analytical mindset early on. The company chose SAS because our solutions can access and combine information from a high number of sources. Gilt also liked the ability to quickly produce reports that were relevant to all facets of its business. I’m happy that the Gilt team recognized our expertise in these areas, but I’m even happier that they’re now taking things to the next level. They’ve scaled up to tackle the more complex analytic challenges, like customizing marketing messages, finding the best customers for cross-sell promotions and helping Gilt's partners understand its shoppers.

I’m glad to say that, thanks to analytics, Gilt is seeing customers browsing in entirely new merchandise categories that it hadn’t previously considered. It’s also seeing new members converting from browsers to shoppers. When analytics was added, the return on investment was positive and immediate.

Gilt tells us they’ve learned three important lessons:

  • Link the data to the business results you want to achieve.
  • Test and measure.
  • Start early with analytics in order to grab the quick wins.

The lesson other retailers can learn from Gilt’s success? Offer a highly personalized customer experience and put analytics at the core. It’s really that simple. And that’s SAS’ gift to retailers. Learn more about the Gilt story. Want to know how other retailers like Macy’s, Chico’s and eBay are using analytics to get big results? Read “3 ways retailers are modernizing with analytics.”

And if you plan to be in New York in January, stop by the SAS booth at Retail’s Big Show 2015, hosted by the National Retail Federation. We’ll be glad to tell you more about how analytics can raise the roof on your retail results. Hope to see you there.

Post a Comment

How do you market to a humbug?

Holiday marketing this year is interesting. With special offers online, competitive shipping rates and more click & collect options, we all know that in-store shopping is no longer the only game in town.

Traditionally, retailers have spent a lot of their marketing budgets trying to draw people into stores on big shopping days like Black Friday, but the best promotions I’m seeing this year are designed to draw more shoppers online.

It makes sense from a business perspective. If you can spread more of the workload and revenues beyond that one day by marketing to consumers with different shopping habits, you don’t have to rely so much on a single shopping day for your holiday forecasts.

But who are these shoppers that might be enticed into spending money online? And what kind of deals are they looking for? We’ve surveyed thousands of consumers to answer those questions – and categorized seven types of shoppers to help you understand their holiday spending habits.


I tend to be a mix between a cybershopper and a last-minute hopeful, but my procrastination can also turn me into a humbug. And I admire the practical shoppers who are already done shopping and now have more time to sit back and enjoy their egg nog.

What about you? Are you a perfect gifter? Or a budget buster? And if you’re a marketer, how might you market differently to each of these customer segments?

Here’s an opportunity: Who’s marketing to the humbugs? It may seem counterintuitive to try to sell to the segment that doesn’t even like shopping, but it’s still 5 percent of the population, and they’re still spending more than $900 each this holiday season. If you could come up with a way to corner the market on the humbugs, you could create a nice niche for your business.

Post a Comment

Entrepreneurs, do you really need an exit strategy?

6855692969_920a16e7c8_zYoung entrepreneurs who master the two-minute elevator pitch know the key points to hit include:

  1. Describing an unmet market need.
  2. Explaining how you're uniquely qualified to meet those needs.
  3. Asking for investment funds to make it happen.
  4. Describing your ultimate plans to sell off the company so investors can recoup their funds.

That last point is called an exit strategy. For better or for worse, it’s become a universal requirement for venture funding and entrepreneurship in recent years.

Think about this, though: What if someone had required Jim Goodnight and his SAS cofounders to plan an exit strategy when they first started SAS back in 1976?

Early plans to sell off the company for a profit would have changed the whole direction of SAS. It would have diminished the company's focus on the customer. It would have undermined the long-term strategy for steady, double-digit growth. And it would have prevented SAS from developing a corporate culture that other companies now seek to emulate.

Investing in evergreen companies

Recently, Tugboat Institute has taken an interest in SAS and other mission-driven organizations that they call “evergreen companies.” According to Tugboat, evergreen companies seek to build sustainable business models, develop long-term goals and provide lasting value for customers.

Recently, we hosted a Tugboat event at our Cary, NC, headquarters where SAS leaders spoke about the benefits and challenges of being an evergreen company. Our advice included:

  • How to grow without becoming complacent. One way we do this at SAS is to encourage collaboration and communication among employees, and to build systems that help keep employees informed and helps them understand the value of their work.
  • How to support continued innovation. You have to develop a culture that celebrates risks within reason. Not every new idea is going to lead to a new product, but you have to develop programs that continue to encourage new ideas.
  • How to find employees that share your same values. Don’t hire anyone that asks about stock options in the first interview. Look for mission-driven employees who share your long-term goals.

About 30 CEOs from various startups attended the event, primarily young entrepreneurs who have started businesses that range from building online communities to selling Christmas trees. One of the CEOs was even a former SAS employee.

It was invigorating. All of the attendees seemed to really care about creating sustainable companies, and not just about becoming gazillionaires in a short amount of time.

One of the goals of the Tugboat Institute is to recognize the benefits of entrepreneurship without an exit strategy. From where I sit, you can see those benefits extending far into the broader communities and the local economies wherever an evergreen company exists.

If you’re an entrepreneur, think about how you might do better with a long-term plan rather than an exit strategy. How could you build a better, more sustainable business if you were in it for the long haul? Visit the Tugboat site to learn more.

Photo by: Ann Hung // license by: cc

Post a Comment

Beyond technology: How we integrate for digital marketing

42-61534963A new report from the CMO Council details the impacts of marketing technology, with a focus on integrated technology strategies. The two most important concepts in the report are easily found highlighted in its title: Quantify how well you unify. 

First, some results from the study that jumped out at me:

  • Companies that have a formal roadmap for digital marketing technology integration and data unification are achieving more targeted, relevant and efficient customer engagements.
  • The most successful companies in the survey have a comprehensive marketing technology strategy and are taking steps to better deploy, manage and integrate their technologies.
  • The most successful companies extend their strategies beyond marketing to include sales, product development, etc., and they generate a significantly higher business impact.

So, unification matters not only in your technology stack, but also in how you approach your marketing strategies. Business and technology goals should be aligned. Likewise, everyone benefits when marketing aligns with sales, product development and other areas of the company.

At SAS, we are seeing strong results from a unified strategy, in terms of being able to create more focused interactions with customers and prospects. Our social media guidelines, for example, are global, and so is our social media platform. And we’re replicating many of our most successful digital marketing programs around the world.

In each of these programs, the importance of measuring our activities cannot be overstated. Any time we can show the impact of a particular program, we find more and more proof that digital marketing makes sense.

You might have to use your data to make hard choices, but ultimately the data makes those choices easier. For example, we have shifted our marketing and advertising budgets so that 70-80 percent of our spend is in the digital space. It’s hard to argue with that shift when the metrics support it.

Beyond unifying technologies and strategies, you also have to unify employees around digital marketing. This can be a challenge when some regions, and some individuals are more digitally inclined than others.

Many creative marketers are still very much focused on traditional marketing activities like brochures, print ads and hotel seminars. When you go 80 percent digital, like we have, your marketing programs require a different skill set. You have to develop training programs and talent that support your new digital programs.

Today, your web site is your first impression in the marketplace. It’s one thing to look pretty on your home page. But you can’t stop there. What are you doing to draw people in? That’s where digital marketing comes in. You have to bring people to your Web site, provide downloadable content and give them reasons to keep coming back.

If you can unify your skill sets and tools around that goal, and do it in a measurable way, you’ll see many of the same benefits that the “top companies” in the CMO Council report are seeing.

Post a Comment

Customer experience is more than just a hot steak

100919562There’s a restaurant here in Raleigh called the Angus Barn.  It’s one of my favorite places to eat and unwind after work. They’ve got top-notch steak and quality cold beer, but what really sets the place apart is the customer experience they provide.

There’s a cigar room in the back called the “Meat Locker.” To get there from the front entrance you have to walk through the kitchen and back hallways of the main restaurant.  It’s like walking through a maze, so half the fun is getting there. You can imagine the delicious aromas the kitchen provides. I’ve been there enough times that I could find my own way through, but each time I arrive, an attendant from the restaurant appears to personally escort me.

What happens next still amazes me.  Every employee I walk by stops what they’re doing, smiles and says, “have a nice dinner,” “enjoy your evening,” or “thank you for being here.” It’s clear they’re all customer driven.  No matter what their job is, they understand how they impact the customer experience, and it keeps me coming back.

What does steak have to do with a software company?  Well, at SAS, we’re customer driven too.  For nearly 40 years, it’s been in the fabric of our culture: We put the customer at the center of our universe. When we make decisions — be it solution offerings or hiring new employees — it’s through the lens of our customers’ point of view.

I’m often asked how SAS continues to build on this cornerstone of our culture.

Here are my top three answers:

  1. Don’t just hear your customers, really listen.  Strive to understand their wants, their needs and their challenges. What help do they need from you? Understand how customers want to interact with you and with their own customers.  That’s what drives business decisions.
  2. Be honest about the help you can provide. This means not automatically saying yes to everything just to satisfy an immediate ask.  This is a tough one, because we want to make our customers happy, but more importantly, we need to offer ideas and do the right things to make them successful in the long run.
  3. Have the right assets in place to aid your customers.  People, processes and technology come to mind.  I love telling folks about our amazing customer contact center, technical support team and newly launched @SAS_Cares social media support team on Twitter.

Much like the staff at Angus Barn, I tell my teams to be out front and ready. We’re all in this together – we all have a role to play in providing a world-class customer experience.

Post a Comment

Can analytics play a role in cybersecurity?

75673483In the same week, my local newspaper ran a story about the hacking of celebrities’ cell phone photos and a separate story about a data breach at 216 neighborhood Jimmy John’s restaurants across the country.

Meanwhile, some of our most trusted global retailers have been the victims of customer data leaks. And governments around the world are struggling to balance the privacy of citizens with the data surveillance needs for preventing legitimate cyber and terrorist threats.

The clear message here is that no one is immune. Cybersecurity is an issue that affects us all.

But what can we do to keep our personal data safe, to ensure our business and transaction data is secure, and to develop surveillance programs within reason to support national security?

In each of those areas – at home, at work and in our national defense – how can analytics play a role?

First, let’s look at the real issues: Networks are being infiltrated in many different ways. From individuals, from computers, from automated systems, from inside and outside trusted boundaries. How can you prevent the threat or recognize the vulnerability when it is happening so quickly and in so many different guises?

And time is of the essence. You can’t wait an hour, a day or a week. If you can’t detect the breach when it’s happening, you don’t have a chance of avoiding potential devastation to your business, your personal reputation or your nation’s security.

What is the solution? We’ve seen that high-speed analytics can detect credit card fraud in the instant that it is happening. And we know how to capture and analyze data as it streams continuously into the network from sensors and devices. Today’s top experts in cybersecurity are combining these concepts to compare normal behavior with abnormal behavior, and to model legitimate traffic so that the system can detect the opposite: suspect behavior.

For instance, should this machine be talking to that machine? Is this frequency of network traffic common for this time of day? Does data passing from this location to that location fit into a larger pattern?

The data is streaming too fast for a human to ask and answer these types of questions. But advanced analytics and the technologies mentioned above make it possible to answer them all within seconds.

When those seconds could be the difference needed to keep your iPhone photos private, to ensure your credit card information is secure and to protect your national interests from cybercrime, I think the answer is clear: Analytics has a definite role to play in cybersecurity.

Post a Comment