The OLAP Council Releases First Benchmark for On-line Analytical Processing Servers
APB-1 Benchmark Provides Quantified/Qualified Measures of Business Intelligence Approach
BOSTON, Mass., Apr. 15, 1996 -- The OLAP Council has released for public comment an Analytical Processing that simulates real-life business situations, measures performance on operations, and quantifies and qualifies the effort to extract business information from existing data. The benchmark measures the performance of on-line analytical processing (OLAP) servers and the servers' ability to respond to complex and changing problems.
Unlike other decision support benchmarks, the simulates a realistic OLAP business situation, in this case sales and marketing applications. Until July 15, 1996, the Council will welcome comments from OLAP users, analysts and other interested parties who may want to make suggestions based on their experience with OLAP technology. The email address to which people may respond is email@example.com.
The OLAP Council, a consortium of sixteen companies, sponsored the benchmark to illustrate the role of OLAP technology within a corporate IT architecture and to help users determine the effectiveness of particular approaches to the multidimensional computations that OLAP servers perform. The is designed to measure the ability of a given server to handle the types of ad hoc queries typically required of an OLAP application.
"The data set has a size and complexity that represents real-world business scenarios," said Bill Carpenter, vice president at Bank of America. "APB-1 is a tool we can use in our evaluation of technology and vendors that can help our company with its specific functional and performance requirements. A critical value of the benchmark is that it measures the amount of effort required to address a problem as well as measuring analytical functions."
"The APB-1 benchmark is precise enough to permit servers available from OLAP vendors to be evaluated on a variety of hardware platforms," said Rick Crandall, OLAP Council spokesman and Comshare chairman. "This benchmark allows a company to evaluate, quantify and qualify its decision-making tools."
The Corporate OLAP View
Changing business conditions are driving new uses of data, and insights into the data are increasingly demanded as the basis for temporary or long-term strategic advantage. Success is often measured with respect to being able to do the total task at all, rather than how fast a particular query returns a component of the insight-generation process. Tools targeting this type of data analysis must take into account the availability of time and skills which are required of data delivery personnel, as well as of the users. The APB-1 benchmark introduces the analytical query time (AQT) metric of performance to quantify the effort required to gain business intelligence from the data source.
In addition to measuring analytical throughput, the requires the disclosure of all programming code used to perform the benchmark. This enables companies to evaluate a given server's ability to respond to complex and changing business issues. Full disclosure allows others to duplicate results using the same documentation and products. Evaluators can judge a server's processing performance and its suitability to an analytical processing task.
The provides a framework for specifying the types and importance of familiar but elusive business questions, and an objective mechanism for comparing results of OLAP servers in a variety of configurations. Individual companies can then assess various costs and benefits of their respective approaches.
The OLAP Council sponsored development of the APB-1 benchmark to provide a means for comparing OLAP servers and measuring the effectiveness and effort required to use particular OLAP servers. The benchmark was written by OLAP-industry veteran Dan Bulos of Symmetry Corp.
The OLAP Council benchmark tests a server's ability to provide a multidimensional view of data by requiring queries of varying complexity and scope. Basic aggregation is performed on some of the dimensions, such as product, customer, and channel. More complex calculations are performed on other dimensions. The "measure" dimension computes ratios and averages. Variances are computed along the "scenario" dimension. A complex model based on historical performance is used to compute the "forecast" scenario.
Analytical processing systems are judged on their ability to create information from data. The APB-1 benchmark contains a representative selection of calculations, both simple and complex. An example of a simple calculation contained in the performance metric is the calculation of margin (sales minus costs). The computation of the forecast is the most complex calculation contained in the benchmark, where historical data is used to project the future and aggregate date is used to estimate input data.
The OLAP Council benchmark contains examples of how time is used in OLAP applications. For example, smoothed sales are computed as a three-month moving average, while inventory is aggregated as an ending balance. The forecast calculation compares this year's vs. last year's knowledge, year-to-date knowledge, and annualization factors.
The OLAP Council is distributing the APB-1 benchmark for public comment for a period of 90 days. The benchmark can be found on the Council's new Website, which can be accessed at http://www.olapcouncil.org.
The OLAP Council
The OLAP Council was founded to provide education about OLAP technology for business intelligence applications and to help position OLAP within a broader information technology architecture. The Council's mission includes sponsoring industry research and working to establish guidelines for OLAP interoperability and data navigation.
The Council, established in January 1995, now includes eleven general members — Arbor Software Corp. of Sunnyvale, Calif.; Business Objects, Inc. of Paris, France and Cupertino, Calif.; Cognos of Burlington, Mass. and Ottawa, Ont.; Comshare, Inc. of Ann Arbor, Mich.; Holistic Systems Ltd. of London; IBM Decision Support Solutions of San Jose, Calif., IRI Software of Waltham, Mass.; Oracle Corp. of Waltham, Mass.; Pilot Software of Cambridge, Mass.; Platinum technology, inc. of Oakbrook Terrace, Ill.; and Planning Sciences International Ltd. of London — and five associate members, KPMG Peat Marwick LLP of Radnor, Penn.; Kenan Technologies of Cambridge, Mass.; Management Science Associates, Inc. of Pittsburgh; TM1 Software (formerly Sinper Corp.) of Warren, N.J.; and Speedware Corp. of Montreal.
Two kinds of membership are available — general and associate — depending on the level of involvement desired. Criteria for all levels of Council membership, in addition to payment of annual dues, include support for the Council's fundamental tenets of on-line analytical processing.
For information about OLAP technology or about membership in the OLAP Council, please access the Council's Website at http://www.olapcouncil.org. For processing membership, call (800) 474-OLAP. Outside the United States, call Rogers Communications at (617) 224-1100.
© 1997 OLAP Council, all rights reserved.