The Top 10 Trends In Data Warehousing
Before the iPhone and Xbox, prior to the first Tweet or Facebook “Like,” and well in advance of tablets and the cloud, there was the data warehouse. For 30 years, businesses have centrally stored data for analysis and data-driven decision making.
For all of that time, the data warehouse has been the business-insights workhorse of enterprise computing. The big trend in the mid 1990’s, the emergence of data warehouses that were a terabyte in size, which at the time was considered a huge amount of data. Today’s leading-edge systems are a thousand times larger—measured in petabytes.
Data warehouses have had staying power because the concept of a central data repository—fed by dozens or hundreds of databases, applications, and other source systems—continues to be the best, most efficient way for companies to get an enterprise-wide view of their customers, supply chains, sales, and operations.
For this reason, businesses that have data warehouses are upgrading and augmenting them with technologies such as Hadoop and in-memory processing, which help with “big data” workloads that are 10 times or 100 times or 1,000 times bigger than before. Meanwhile, businesses that have relied on piecemeal data-analysis solutions in the past are now establishing data warehouses to get a more complete picture of the enterprise.
In other words, data warehouses aren’t just bigger than a generation ago; they’re faster, support new data types, serve a wider range of business-critical functions, and are capable of providing actionable insights to anyone in the enterprise at any time or place. All of which makes the modern data warehouse more important than ever to business agility, innovation, and competitive advantage.
A new white paper from Oracle explores the top 10 trends and opportunities in data warehousing. Here’s a recap of that Top 10 list along with my own take on each trend.
- The “datafication” of the enterprise requires more capable data warehouses. Mobile devices, social media traffic, networked sensors (i.e. the Internet of Things), and other sources are generating a growing stream of data—some would say a fire hose of data. IT teams are responding by adding new capabilities to data warehouses so they can handle new types of data, more data, and do so faster than ever.
- Physical and logical consolidation help reduce costs. The answer to datafication isn’t to throw more money at these systems. Or put another way, ten times the data shouldn’t translate into ten times the cost. So burgeoning data warehouses must be consolidated, through a combination of virtualization, compression, multi-tenant databases, and servers that are engineered to handle much larger data volumes and workloads.
- Hadoop optimizes data warehouse environments. The open source Hadoop program, with its distributed file system (HDFS) and parallel MapReduce paradigm, excels at processing very large data sets. That makes Hadoop a great companion to “standard” data warehouses and explains why a growing number of data warehouse managers are now using Hadoop to shoulder some of the heaviest workloads.
- Customer experience (CX) strategies use real-time analytics to improve marketing campaigns. Data warehouses play a pivotal role in CX initiatives because they house the data used to establish a comprehensive, 360-degree view of your customer base. A data warehouse of customer information can be used for sentiment analysis, personalization, marketing automation, sales, and customer service.
- Engineered systems are becoming a preferred approach for large scale information management. If you’re not careful, data warehouses can become a complex assembly of disparate pieces—servers, storage, database software, and other components—but that doesn’t have to be the case. Engineered systems such as Oracle Big Data Appliance and Oracle Exadata Database Machine are preconfigured and optimized for specific kinds of workloads, delivering the highest levels of performance without the integration and configuration headaches.
- On-demand analytics environments meet the growing demand for rapid prototyping and information discovery. If you’re familiar with cloud computing’s software-as-a-service model, then you’ll appreciate the concept of “analytics as a service.” Technical breakthroughs such as Oracle Database 12c’s pluggable database feature make it easy for administrators to provide “sandboxes” in a data warehouse environment for use in support of new analytics projects.
- Data compression enables higher-volume, higher-value analytics. The best way to counter non-stop data expansion is—what else?—data compression. Your organization’s data may be growing at 10X, but advanced compression methods, such as Oracle’s Hybrid Columnar Compression, can match that. Using compression, companies can capture and store more valuable data, and they can do it without 10X the cost and 10X the pain.
- In-database analytics simplify analysis. Ideally, your data warehouse will have a range of ready-to-use tools—native SQL, integration with the R programming language, and data mining algorithms, for example–to jump start and facilitate data analysis. These kinds of in-database analytics capabilities minimize the need to move data back and forth to other systems and applications for analysis, resulting in more streamlined and optimized data discovery.
- In-memory technologies supercharge performance. The emergence of in-memory database architecture brings race car-like performance to data warehouses. The term in memory is highly descriptive, of course. It refers to the ability to process large data sets in system RAM, accelerating number-crunching and reporting of actionable information.
- Data warehouses are more critical than ever to business operations. While it’s true that data warehouses have been around for years, their value keeps growing because they represent a company’s crown jewels—prized data on customers and business performance. And organizations are finding new applications for data warehouses, as described in the example above, where healthcare providers are using enterprise data warehouses to improve patient care and streamline operations.
Taken together, these Top 10 trends describe a new generation of data warehouses that are bigger, better, and faster than ever before, transforming data into information and information into actionable insights, enabling businesses to forge ahead with unprecedented speed and agility.