Big data research explains spicy curry and a thrilling novel

FoodLike all scientific breakthroughs, there needs to be some sort of experiment or evidence gathering to prove a hypothesis. Sometimes these breakthroughs are unrelated to the original hypothesis and are made by accident - as long as there’s some form of information to analyse, there’s scope for discovery. With so much of an ordinary person’s life now open to analysis by the data they leave behind, we are beginning to make breakthroughs that explain everyday life experiences.

Recently a piece in The Hindu reported a study that used data analytics techniques to establish an unusual feature of Indian cuisine. It found that, whereas most other global cuisines rely on positive food pairing – the pairing of similarly flavoured ingredients - Indian cuisine instead relies on negative food pairings using dissimilarly flavoured ingredients. They also discovered, by shuffling around ingredients in a recipe to observe its effect on negative food pairing, that it was the spice that drove the negative pairing. Of the top 10 ingredients whose presence biased the flavour-sharing pattern of Indian cuisine towards negative pairing, nine were spices.

Indian cuisine is much more complex as 20 ingredients may be used for a dish compared to, say, five for a typical Western dish. When you consider this, and the variation in Indian cuisine across regions and groups, there are multiple possible combinations of ingredients which may all get a different reaction depending on who is tasting them. It demonstrates how Indian recipes could be personalised according to what spice combinations a person prefers. Read More »

Post a Comment

The tale of two innovation labs for big data

468837845Recently I have been out speaking with a number of organizations about the idea of the innovation lab concept , which I discussed in a previous blog post, as the way to unleash the power of big data and make even the largest of companies as agile as a startup. During my discussions there are a couple of things that I am observing, that I wanted to share with you, since it seems there are different types of innovation labs in organizations:

  1. Many companies have something I am going to label an IT innovation lab where they are experimenting with "big data" technologies. These IT innovation labs are NOT the same as the “data related” innovation lab that companies need to put in place to remain agile in this new digital world. The focus of the IT innovation lab is to test the technology and its integration whereas the focus of the "data related" innovation lab is to test hypothesis around mashups of data and different analytical approaches In the digital world, information gleaned from data is your best competitive weapon, and speed is a critical component to your success.  It is my opinion these should be separate and distinct in a companies strategy as each has a role to play. This post will focus on the tale of two labs and how they differ.
  2. Tight budgets, and the significantly different focus of the “data focused” innovation lab, are causing organizations to ask for support to obtain funding given it is generally a new concept to not have a concrete business problem solved as part of asking for an investment. A second post in this series will focus on how I suggest organizations can build the business case for the data focused innovation lab which I believe is vital to the future success of all organizations no matter how large or small they are.

Read More »

Post a Comment

Six analytics lessons from the automotive roundtable

Increasingly, automotive executives want to talk about the "Art of the Possible" in analytics. So we took the opportunity to invite leaders around the industry to an Automotive Analytics Executive Roundtable to share their stories and spark new ideas. A myriad of diverse speakers covered a variety of topics on big data/Hadoop, the connected vehicle, customer experience, building a culture of analytics, and innovative retail analytics.

To open the day, we discussed how analytics can address key issues facing the automotive industry. One topic was the use of analytics to address changing consumer access and rise of consumer voice and influence through online channels which enable better insight to product design and quality. With global manufacturing supply chains, the challenges and opportunities to adequately plan, forecast and optimize inventory and pricing are numerous. As we looked at "connected everything," we explored use cases for the connected vehicle, factory, dealer and consumer. None of these are possible without the use of big data analytics. Keep reading for my top six lessons from the day that I hope you can use too: Read More »

Post a Comment

What can we learn from the world's largest repository of international trade data?

Did you know the US-Malawi trade relationship is based almost entirely on tobacco? Or that apparel & accessories drive Morocco & Tunisia exports while their North African neighbors rely on oil?

Trade balance report showing significance of tobacco to US-Malawi trade (click to enlarge)

Trade balance report showing significance of tobacco to US-Malawi trade (click to enlarge)

The United Nations Statistics Division collects detailed international trade data from more than 200 countries on the import and export of hundreds of commodities. Talk about Big Data... Until now, the UN Comtrade database has been too large for most people or organizations to consume and analyze in its entirety.  Thanks to a new collaboration with SAS, the data is now available to everyone, via a web browser or mobile tablet, and unlocks valuable information that benefits policy makers, businesses, researchers and the general public. A few of the many possible use cases include:

  • A trade minister can interrogate decades of international trade data on a mobile tablet.
  • A global business can access immediate insights on risks and business performance in local markets
  • A university student can gain the valuable experience of mining millions of rows of data for hidden trends and stories

SAS Visual Analytics for UN Comtrade brings Big Data to the masses, enabling anyone to glean insights from the most comprehensive collection of international trade data in the world...300+ million rows.  Employing high performance data visualization, big data and cloud computing technologies, this online service exposes stories hidden across hundreds of trading partners and thousands of commodities to reveal how nations have interacted economically through the last three decades.

What stories can you uncover? How much oil does your country import? How much oil do your trade partners say they export to you? Is there a disparity? The mirror statistics raise plenty of questions.

Users can explore the date through several views, including:

  • Imports/exports: Top importers and exporters by world, region and country, and what commodities they trade.
  • Trade balance: Top commodities bought and sold on a global or country level.
  • Trade composition: Most frequently traded commodities by partners.
  • Mirror statistics: A comparison of import/export data, as reported by partners on both sides of a trade relationship.
  • Trade history: Top trading partners for any given country, with an animated bubble plot tracking relationships over the time.
  • Data: All data, presented in tabular format with powerful filters
  • Historical analysis: See trade history across any combination of partner(s), commodities and year(s)
Users can drill into data on the top importers and commodities (click to enlarge)

Users can drill into data on the top importers and commodities (click to enlarge)

SAS offers many of these unparalleled interactive visualizations for free at the link above. It is our contribution in advancing the global data revolution. We’re not just making the data public, we’re making the insights public – for the good of society.

In my next blog post, I will share lessons learned from this journey with the UN to help with their Big Data visualization initiatives.

What can you learn from the data? What surprised you? Please share your insights!

Post a Comment

The threat from within: battling internal fraud

WalletFraud is a growing problem for businesses – and one of the biggest threats comes from an organisation’s own employees. In many countries, the incidence of internal fraud is rising. According to the Credit Industry Fraud Avoidance System (CIFAS), in the UK alone there was an 18 percent rise in the total number of staff frauds recorded in 2013 when compared to 2012.

It is a problem that differs from territory to territory. PwC’s 2014 Global Economic Crime Survey revealed that South African organisations suffer “significantly more procurement fraud, human resources fraud, bribery and financial statement fraud than organisations globally.” Equally, according to CIFAS’s Employee Fraudscape report, published in April 2014, the number of unsuccessful employment application frauds in the UK increased by over 70% in 2013 compared with 2012.

The problem is becoming a priority for many organisations - but the main area of focus differs from country to country. The spectre of financial loss is critical everywhere - but in many places it’s outweighed by the fear of reputational damage. In the UK and the US, where we have recently seen multiple market abuse and unauthorised trading cases hitting the headlines, there is a strong emphasis on addressing regulatory requirements.

Read More »

Post a Comment

When the going gets tough, the tough use analytics

When do analytics really provide value? All the time, of course. However, one of the best times for analytics to prove their value is when you are asked to do more with less.   Often, the reason we are asked to do more with less is because of an economic downturn for our company, industry, or the economy as a whole. It is ironic that investing in analytics, which could have a meaningful impact on how well our organization functions long-term, is sometimes only considered when times are good and resources are plentiful, not when times are tough.  However history shows us time and again that analytics can help you the most by allowing you to do more with less.

Faster analytics provide even more value

High perCompress_Timeformance analytics makes it even easier to do more with less (especially in less time).  As you know, time is a limited resource that we just can't get back and for many decisions, timing is crucial. If it takes too long to get analytic insight needed to make a decision, it doesn't matter how great that insight is. When it's too late, it's too late.

The value that high-performance analytics adds on top of the better insights you get from predictive and prescriptive analytics is the ability to "compress time" and deliver those insights quicker, thereby improving their value to the overall business decision process already in place.

The history of fast analytics

If you think about it, this is really why computers were first invented: to deliver insight faster than previously thought possible.   Alan Turing developed the Turing machine, considered to be a model for the general purpose computer, to speed up the process of decoding the German military encrypted messages created by their Enigma machines in World War II.  The use of mathematics to help recognize patterns in these messages (analytics) coupled with high performance (computer processing of the Turing Machine) means that the earliest form of high-performance analytics helped bring about the defeat of the Axis powers in World War II.  Another fascinating example of analytics helping the allies win WW II can be found in Jordan Ellenberg's "How Not To Be Wrong - The Power of Mathematical Thinking".  In it you can read how Abraham Wald, a member of the Statistical Research Group (SRG), a (at the time) classified program tasked with helping the war effort, solved the issue of where best to add more armor on airplanes to decrease the number of planes being shot down.

Talk about analytics having a meaningful impact!

Imagine how you might apply analytics to help your organization solve problems or improve efficiency.    Bob Dudley, BP Group Chief Executive, provides a potential case in point in Oil and Gas with this statement from his speech he gave earlier this March at the Mexican Energy Reform Summit 2015. "If the global energy environment was highly competitive before - at $100 a barrel - it just got ultra-competitive at $50 to $60 a barrel."

This is a perfect example of where analytics can have a big impact for the oil & gas upstream processes. By helping to reduce the overall costs with getting oil out of the ground and then improving the processes of getting it to market, companies can potentially improve profits in this down market.   Read more about how analytics can help reduce costs in upstream exploration and production(E&P) in my previous post on "Analytics is an enhanced oil recovery process" and Keith Holdaway's book, "Harness Oil and Gas Big Data with Analytics." Interested in how analytics can help in the downstream process then see my colleague, Charlie Chase's recent post entitled "How to use analytics with O&G downstream data to improve forecast accuracy."

Post a Comment

Using analytics to detect product defects

Companies have become obsessed with product quality – and for good reason. Customer dynamics, their willingness to expose product quality issues socially on the web, and the ease with which they can jump to competitive products make quality a more important differentiator than ever before.

Over the past 15 years,  I have had the opportunity to work with many organizations, across industries, to help them leverage analytics to improve the quality of their products and services. With such high stakes, it is always a bit of a surprise to hear how exposed many companies are when it comes to detecting and correcting product defects in the field. For too many, they are entirely reliant on basic business intelligence reports and Excel.  They know when a claim occurs and can slice and dice claim counts to, at best, infer the existence of a possible defect. The result is they spend more time chasing false signals, while real defects continue to plague their customers.

Time is money – and brand value.  The longer issues linger, the more they cost the manufacturer, and the more negative pressure is piled on the brand.  Advanced analytics can drive significant value by helping organizations greatly reduce the time and effort involved in identifying and resolving product defects that make it to the field.  Here are three ways analytics can drive value in reducing the impact of product defects: Read More »

Post a Comment

Are you using downstream data to improve forecast accuracy?

Downstream data have been electronically available on a weekly basis since the late 1980s. But most companies have been slow to adopt downstream data for planning and forecasting purposes. Let's look at why that is.

Downstream data is data that originates downstream on the demand side of the value chain. Examples include retailer point-of-sale (POS) data, syndicated scanner data from Nielsen, Information Resources Inc. data and Intercontinental Marketing Services (IMS) data.

Prior to electronically available downstream data, manufacturers received this type of data in hard copy format as paper decks (after a 4-6 week lag). Once received, the data points were entered into mainframes manually via a dumb terminal (a display monitor that has no processing capabilities; it is simply an output device that accepts data from the CPU).

In fact, downstream data have been available to consumer products manufacturers for several decades. Subsequently, the quality, coverage and latency of downstream data have improved significantly, particularly over the past 20 years, with the introduction of universal product barcodes (UPC) and retail store scanners.

Today, for many companies, data management capabilities have advanced so quickly that the challenge now is how to report and make practical use of it all.

Data storage costs have fallen significantly over the past decade. Sales transaction data is being captured at increasingly granular levels across markets, channels, brands and product configurations. Faster in-memory processing is making it possible to run simulations in minutes that previously had to be left to run overnight.

In fact, companies receive daily and weekly retailer POS data down to the SKU/UPC level through electronic data interchange transfers (electronic communication method that provides standards for exchanging data via any electronic means). These frequent data points can be supplemented with syndicated scanner data across multiple channels (retail grocery, mass merchandiser, drug, wholesale club, liquor, and others) with minimal latency (1-2 week lag) by demographic market area, channel, key account (retail chain), brand, product group, product, and SKU/UPC.

Consequently, downstream data are the closest source of consumer demand above any other data, including customer orders, sales orders, and shipments. Unfortunately, most companies primarily use downstream data in pockets to improve sales reporting, uncover consumer insights, measure their market mix performance, conduct price sensitivity analysis, and gauge sell-through rates.However, no manufacturer including consumer packaged goods (CPG) companies have designed and implemented an end-to-end value supply chain network to fully utilize downstream data.

Read More »

Post a Comment

It’s the Internet of connected life at Mobile World Congress

ATT MWCThe Internet of Things is coming fast and furious. We clearly know what these “things” are, and were able to see prototypes at last week’s Mobile World Congress (MWC) which hosted some 93,000 attendees.

Things = connected life = cars, homes (thermostats, washer and dryers, vacuum cleaners, security systems, refrigerators, etc.), toothbrushes – yes, I said toothbrushes, bicycles, loading docs, shipping ports, airports, wearables, trash cans, plus our traditional devices like the smartphones and tablets and the list grows!

AT&T was at the forefront of IoT innovation at MWC providing virtual reality tours of connected cars, a loading dock, shipping containers, farms, and the connected home service that they now provide in the US. All of this is fascinating when we consider where we were just five to ten years ago. Read More »

Post a Comment

Internet of Things: The industrial internet

Today in manufacturing there has been a lot of investment in automation, supervisory controls, quality, and execution systems. The amount of data produced and now being captured is staggering.  The data captured in industry will re-define what is “big” in big data.

Yet, for all this investment:

  • Equipment still fails.
  • Scrap is still produced.
  • Safety incidents still occur.

Therefore, there are things about manufacturing processes we still don’t know.

Read More »

Post a Comment