Oil and gas data management overview

Black background with shiny squaresIn the oil and gas industry, analytics are used to improve both upstream and downstream operations, from optimizing exploration and forecasting production to reducing commodity trading risk and understanding customer's energy needs.

If you plan to derive value from the digital oil field, big data, and analytics, one of the first things you'll need is a proper data management strategy.

Your oil and gas data management strategy should consider data from existing systems as well as new sources, including all the sensors being added to equipment used in the upstream as well as the downstream areas of the oil and gas industry.

With data flowing in from exploration, drilling, production and usage meters, the rapid growth of data sources presents a data management challenge for the industry.

Consider this excerpt from Chapter 2 of Keith Holdaway's book, Harness Oil and Gas Big Data with Analytics:

Oil and gas operators are faced with a daunting challenge as they strive to
collate the raw data that serve as the very foundation of their business success, transforming that raw data into actionable knowledge. However, with the exponential growth in data volumes and the breadth of siloed, disparate data sources increasing at ever-faster rates, the industry is realizing that data management is fundamental to their success.

Even as data volumes grow, it's important to remember that data management isn't just about storing larger and larger amounts of data but figuring out what data is most relevant for the problems being explored, and then processing this data in a timely manner. Depending on your analytics needs, that processing might involve simple queries or more intensive resources like analytic workloads with more iterations to produce more informative results.

Data management involves handling streaming data, near real-time data, data stored on disk, and data stored in archive systems and being able to integrate all or some of these together in the best formats for analytics and reporting.

It's important to incorporate predictive analytics in some or all of the data gathering or data reporting points.

For example, event stream processing is part of most modern data management strategies because it  allows you to apply if-then type rules to data in stream. It also applies analytics to your sensor data in stream, and take actions with that data as a result of the activity in the data.

Data quality capabilities should also be applied to clean up messy or missing data so decision makers receive quality information that helps them make better decisions in running your organization.

For more details on these topics, please read our whitepaper, Analytic Innovations Address New Challenges in the Oil and Gas Industry.

Post a Comment

Measuring the success of sustainable organizations

Book cover for The Sustainable OrganisationLooking at the top 100 organizations in the world, have you ever wondered which ones we’d really miss if they were to disappear? Give it a try. I bet you don’t choose the ones with the highest profits. You probably make your picks based on meaning. You probably chose the organizations that give us something we truly need, or add to our health and happiness, or take special care to protect resources for future generations.

I started asking these types of questions when I became a family man. Thinking of the future awaiting my children and grandchildren, I grew alarmed by the increasing levels of instability and insecurity in the world today. I wondered why we’re so willing to shortchange the future for the sake of immediate gain, and I set about looking for a way to assess who the good guys are. Who’s doing things differently, and what impact are they having?

Read More »

Post a Comment

Education meets big data: implement, improve and expand your SLDS

In my previous blog post, I discussed the benefits of a Statewide Longitudinal Data System (SLDS) and shared a SAS book on the subject: Implement, Improve and Expand Your Statewide Longitudinal Data System by Armistead W Sapp III and Jamie McQuiggan.

Implement, Improve and Expand your Statewide Longitudinal Data System

Today, I'm sharing a conversation I had with one of the book’s authors, Armistead Sapp. In it, we discuss state funding, big data and overcoming challenges with patience and persistence.

What was the initial impetus for writing the book and how has it been received?

Armistead Sapp: We wrote the book to address the questions that states might have with implementing, improving and expanding their SLDSs. When I first starting write this, I envisioned we would be talking to the state education leaders and the folks in charge of the SLDS in each state.

As the federal grant program expanded, we realized there was an abundance of funding but there was also a lack of direction. Even after several rounds of funding, states were still struggling. Some states were doing things well and some not so well.

When I started working with the State of North Carolina, on their SLDS at SAS, I realized that there were questions that every state needed to answer. So, I thought it would be important to write a book to share the best practices we were seeing by working with the states on these projects.

Read More »

Post a Comment

Analytics and Hadoop partnering for success

Alan Saldich, VP of Marketing at Cloudera

Alan Saldich, VP of Marketing at Cloudera, discusses how SAS and Cloudera are tackling Cybersecurity

In a complicated, fast-paced and connected world, you don’t succeed alone.  SAS and Cloudera have a successful  partnership that dates back several years. Our products are complementary and provide significant quantifiable value to customers who run them on the same cluster.  Add Intel to the mix and you have a trio of success as evidenced at the recent Analytics + Hadoop Event in New York featuring SAS, Cloudera and Intel.

Cloudera was the first commercial distributor of Hadoop; It enables SAS analysts to access a unified (and essentially unlimited) set of data - structured, unstructured, new, legacy - using familiar tools and frameworks.

Companies are looking to modernize their analytics with Hadoop at the core and to capitalize on the myriad ways to extract value from their data.  The process itself is relatively straightforward: Read More »

Post a Comment

If you don’t buy a ticket to the data lottery, your competitors will

LotteryYou have to be "in it to win it" as they say. This is becoming the case for many organisations that need to start using data to make better, evidence-based business decisions. Today, using analytics is not so much a data lottery as a data necessity.

Some businesses may not have embraced analytics at all, while others may not be applying it across all aspects of the business, or may be in need of a modernisation programme to bring them up to speed with competitors.

Read More »

Post a Comment

Who you gonna call (for a cybersecurity issue)?

Man holding up an old fashioned phone During a lighthearted moment in a serious conversation, Howard Schmidt, cyber security advisor to multiple presidents, told a Wall Street Journal interviewer that relying on a government agency as your primary backstop during a major cyber security breach is akin to calling Ghostbusters: you might not get the help you need when you need it.

Joking aside, the question of whom to call was a real one, posed to a group of CEOs during a cyber-attack simulation exercise. Unfortunately none of the group could answer with certainty. In fairness to the CEOs, “who you gonna call” is a loaded question because the agencies themselves lack clarity on this issue.

Read More »

Post a Comment

From strategic to operational decision making: Decisions at scale

Man in suit overlooking city landscapeLuckily, or perhaps better said, hopefully, we only need to make the big life decisions every now and then. What school to go to? Who to marry? What job to take? Where to live? There’s no penultimate answer to these decisions, but we all take them to the best of our knowledge, feeling and ability.

Likewise in organizations, we don’t make big strategic decisions every day. Which customers to focus on? Which products? Which regions? Strategy revolves around crunching the numbers, evaluating the situation, incorporating past experience, and choosing a direction. Since emerging on the organizational agenda, analytics have always played a role in informing strategic decisions.

Read More »

Post a Comment

Communications from the final frontier

Pluto as seen by New Horizons

Pluto photographed by New Horizons

When I was a kid learning about the solar system and building those models out of hastily-painted styrofoam balls of varying sizes, Pluto was a planet. A full-fledged, legitimate planet just like the other eight. But In August of 2006, just 7 months after NASA launched the New Horizons Mission to explore Pluto and beyond, the International Astronomical Union designated Pluto a “dwarf planet”—putting a highly controversial end to its 76-year-old claim as the furthest planet from the sun.

Prior to the New Horizons Mission, the most powerful telescopes we had were only able to show Pluto as a blurry disk. Decades of scientific speculation had determined it to be an icy rock. Our knowledge about this astronomical anomaly was very limited—after all, it’s 3 billion miles away from Earth. Nine years later, New Horizons has made that long journey and is finally reporting back, giving us a glimpse of what this mysterious icy mass and its surroundings really look like.

The probe made its closest Pluto flyby on Tuesday of this week. And on Wednesday, we already had photos. That may not seem impressive until you realize just how far that data traveled to get back to us.

Read More »

Post a Comment

Machine learning in action today

What's that productivity related quote by Charles Dickens?

"My advice is never do tomorrow what you can do today."

For years, machine learning has been written about and discussed widely with a focus on the benefits it will bring in the near future. But guess what? The future for machine learning is now.

The ability run a model tournament across a wide variety of analytical models, to provide hundreds or even thousands of the best predictive models for each segment of your choice and then take action is available to you today. The fusion of machine learning with data mining has made this possible today.

Read More »

Post a Comment

Why the Internet of Things will change how you interact with customers

Blurry people walking on city streetThere's a lot of talk right now about the Internet of Things and how it's likely the prime catalyst for the digital transformation of organizations over the next few years. Billions of sensors, and devices with sensors, all generating data in a hyper-connected world where it can be easily shared or transmitted – what could possibly go wrong?

While the Internet of Things holds out promises to revolutionize healthcare, improve energy consumption and more, the area where it will touch most of our lives first is in how we, as consumers, engage with brands.

This is why you need to understand the importance of the Internet of Things beyond the hype. It will change the way you interact with customers in a market that will be differentiated by digital services and unique user experiences.

Read More »

Post a Comment