olap « WordPress.com Tag Feed

Apakah OLAP, MOLAP, ROLAP dan HOLAP itu?

Ardijan Abu Hanifah — Wed, 25 Nov 2009 03:59:25 +0000

OLAP (On-Line Analytical Processing) adalah Database Multidimensional pada sistem analisa data tingkat lanjut yang mendukung pengambilan keputusan, bisnis model, dan aktifitas riset.

OLAP menyediakan cara untuk menampilkan data Multi Dimensional yang ada dalam Data Mart atau Data Warehouse, dengan OLAP dapat dibuat Cube yang mengorganisasikan data dan membuat summary data untuk query yang effisien.

Karakteristik OLAP
- Menggunakan teknik analisa data Multidimensional
- Menyediakan dukungan database tingkat lanjut
- Menyediakan cara pakai yang mudah dan User Interface yang
mudah difahami.
- Mendukung arsitektur Client/Server

OLAP Features

Beberapa produk OLAP secara logika adalah sama, kesamaan tersebut antara lain dalam hal-hal berikut:
- Berdasarkan Dimensional
- Pre-Aggregate untuk meningkatkan performance dan
- Mendukung bahasa analisa data tingkat lanjut

Sebenarnya secara fisik ada perbedaan pada cara menyimpan Cube dalam Data Warehouse, dan dipasaran dikelompokkan menjadi seperti berikut:

MOLAP
Multidimensional OLAP, Data disimpan dalam bentuk Multidimensional Database.

Pros:
– Performance hebat, karena MOLAP memang dibangun
untuk pengambilan data yang cepat, dan optimal untuk
operasi Slicing dan Dicing.
– Dapat membentuk kalkulasi yang komplek dan cepat.
Semua kalkulasi telah dihitung saat Cube dibentuk.

Cons:
– Jumlah volume data yang dapat ditangani terbatas. Karena
semua kalkulasi telah dihitung saat Cube dibentuk maka
untuk menyimpan hasil kalkulasi tersebut diperlukan
volume data yang besar dalam Cube nya sendiri.
– Diperlukan investasi tambahan karena teknologi MOLAP
Cube seringkali belum dimiliki oleh organisasi, dengan
kata lain untuk mengadopsi teknologi MOLAP ada peluang
untuk menambah investasi tenaga dan biaya.

ROLAP
Relational OLAP, menggunakan Relational Database baik untuk menyimpan Detail data maupun untuk menyimpan Aggregate nya. Memanage pembuatan dan perawatan Aggregate.

Pros:
– Dapat menangani jumbalh volume data yang sangat besar,
batasan ukuran volume data yang ditangani pada
teknologi ROLAP adalah batas dari volume dari Relational
Database yang dipakai. Dengan kata lain pada ROLAP
sendiri tidak ada batasan volume data.
– Dapat memanfaatkan fungsi-fungsi yang ada pada
Relational Database yang dipakai.

Cons:
– Performance dapat lambat, karena setiap ROLAP report
pada dasarnya adalah SQL Query pada Relational Database,
waktu Query dapat lebih lama jika volume data semakin
besar.
– Fungsi SQL yang terbatas, karena teknologi ROLAP
terutama tergantung pada pembentukan statement Query
pada Relational Database, dan tidak semua kebutuhan
dapat terpenuhi dengan SQL Statement. ROLAP vendor
telah mengantisipasi resiko ini dengan cara membuat Tool
out-of-the-box untuk fungsi-fungsi yang kompleks bahkan
memungkinkan user untuk mendefinisikan fungsi-fungsi
yang dibutuhkannya sendiri.

HOLAP
Hybrid OLAP, Menggabungkan kedua teknologi diatas. HOLAP menggunakan Relational Database untuk menyimpan Detail data dan menggunakan Multidimensional Database untuk menyimpan Aggregate nya.

HOLAP menggabungkan kelebihan-kelebihan yang ada pada MOLAP dan ROLAP.
HOLAP juga memanfaatkan teknologi MOLAP cube untuk mendapatkan performance yang lebih cepat.
Jika dibutuhkan informasi di level detail, HOLAP dapat ‘Drill-through” dari Cube masuk kedalam Relational Database yang mendasarinya.

OLAP Comparizon

Reff:
-Ponniah, Paulraj, Data Warehousing Fundamental,New
York-USA,2001.
Jhon Wiley & Sons, Inc.
-Thomsen, Erik, 1999. Microsoft OLAP Solution, New York-USA,
Jhon Wiley & Sons, Inc.
-Peterson, Timothy, Microsoft OLAP, Sams Publishing, USA, 2000

VIBlend announces new Pivot Grid for Silverlight

viblend — Wed, 25 Nov 2009 03:43:58 +0000

VIBlend is proud to announce the release of a new OLAP Data Grid Control for Silverlight

VIBlend DataGrid for Silverlight is the first grid control that brings together the functionality of traditional data grids, hierarchical grids and pivot tables. This feature complete control includes a built-in OLAP engine which can easily process data in tabular format and aggregate it based on user defined criteria.

Getting started with VIBlend DataGrid for Silverlight is easy and it takes very little code to implement complex scenarios from end to end.

VIBlend DataGrid for Silverlight is one of the fastest Silverlight grid controls and can easily work with large number of rows.

VIBlend Silverlight Controls are written entirely in C# with the latest .NET technologies.

To learn more and download your free trial copy of VIBlend Silverlight Controls, visit http://www.viblend.com

Apakah Data Warehouse itu?

Ardijan Abu Hanifah — Fri, 20 Nov 2009 09:30:15 +0000

Data Warehouse adalah relasional database dengan desain tertentu yang memudahkan Query dan Analisis.

Pada umumnya Data Warehouse berisi data sejarah yang diturunkan dari data transaksi dan beberapa data dari sumber data yang lain.

Dengan Data Warehouse maka Organisasi bisa mendapatkan data yang sudah terkonsolidasi dari berbagai sumber data.

William Harvey Inmon atau biasa dipanggil Bill Inmon pada tahun 1991 dalam bukunya “Building the Data Warehouse”
mendefinisikan Data Warehouse sbb.:

“A data warehouse is a subject-oriented, integrated,
timevariant, and nonvolatile collection of data
in support of management’s decision-making
process.”

Subject-oriented:
- Data di kumpulkan berdasarkan subjek utama, misalnya
Customer, Product atau Sales.
- Data Model dan analisa diutamakan untuk pengambil
keputusan bukan untuk proses transaksi sehari-hari.
- Menyediakan cara yang sederhana dan ringkas untuk
menampilkan data sekitar subjek tertentu dengan cara
mengeluarkan data yang tidak berguna dalam proses
pengambilan keputusan.

Integrated:
- Dibangun dari integrasi berbagai macam sumber data
misalnya Relational Database, Flat Files, Online
Tansaction record, dsb.
- Menggunakan teknologi Data Cleansing dan Data
Integration. Tujuannya:
- Memastikan konsistensi Naming Convention,
Encoding Structure, Ukuran attributes. dsb.
diantara sumber data yang berbeda-beda.
- Ketika data dimasukkan ke Data
Warehouse, data tesebut telah dikonversikan.

Time Variant:
- Rentang waktu data untuk Data Warehouse pasti lebih
panjang jika dibandingkan dengan sistem operasional
- Data Sistim Operasional : Data saat ini
- Data Data Warehouse : Menyediakan informasi
dari perspektif sejarah misal 5 s/d 10 tahun
- Setiap struktur Key dalam Data Warehouse selalu
mengandung elemen Time secara implisit atau explisit
- Namun pada Key data operasional mungkin tidak
mengandung elemen Time.

Non Volatile:
- Artinya begitu data masuk ke Data Warehouse
seharusnya Data tidak boleh berubah lagi.
- Secara fisik data pada Data Warehouse disimpan
terpisah dari operasional sistem.
- Operasi Update tidak pernah terjadi pada lingkungan
data Warehouse, karena
- Tidak perlu melakukan proses transaksi, Recovery,
dan mekanisme kontrol proses simultan.
- Hanya memerlukan dua operasi proses Data yaitu
Data Loading dan Data Retrieving.

Elemen-elemen dasar Data Warehouse

Source System
Operasional sistem yang berfungsi mencatat transaksi dari
suatu bisnis. Source System biasa disebut juga sebagai
Legacy System.

Data Staging Area
Tempat penyimpanan data sementara, untuk melakukan
proses Clean, Transform, Combine, Deduplicate,
Household, archive, dan menyiapkan sumber data untuk
digunakan Data Warehouse.

Staging Area tidak diperuntukkan sebagai sumber data
untuk reporting karena datanya masih kotor dan bersifat
sementara.

Presentation Server
Target mesin yang secara fisik adalah mesin dimana data
dari Data Warehouse di organisasikan dan disimpan untuk
dilakukan query oleh End User, Reporting dan aplikasi yang
lain. Disini data disimpan dengan framework Dimmensional.

Jika menggunakan Relational Database maka tabel akan
diorganisasikan dalam bentuk Star-Schema, namun
jika tidak menggunakan Relational Database maka akan
disimpan dalam bentuk OLAP.
(mengenai OLAP akan dibahas pada sesi tersendiri)

Dimensional Model
Data Modeling yang khusus dipakai pada Data Warehouse,
sebagai alternatif lain dari E/R Modeling yang biasanya
dipakai pada sistem transaksional.
(mengenai Dimensional Modeling, akan dibahas pada sesi
tersendiri)

Business Process
Dalam terminologi disini bukanlah Business Process seperti
pada transaksional sistem, tetapi lebih tepat maksudnya
adalah mengelompokkan sumber data menurut kesamaan
temanya. Misalnya Finance, Marketing, Production. etc.

Nantinya setiap Business Process akan diimplementasikan
sebagai Data Marts.

Data Mart
Potongan secara logika dari Data Warehouse secara
keseluruhan.

Data Mart bisa juga dilihat sebagai potongan secara logika
dari data Warehouse terhadap suatu Business Process
tertentu.
(mengenai Data Mart, akan dibahas pada sesi tersendiri)

Data Warehouse
Adalah sumber data yang bisa di Query dalam suatu
perusahaan. Sebenarnya Data Warehouse adalah
gabungan dari beberapa Data Marts.

Operational Data Store
Pada awalnya ODS adalah titik integrasi dari berbagai
operasional sistem. Karena ODS juga mendukung access
operasional dan bisa di update maka ODS seharusnya
ditempatkan diluar Data Warehouse.

Pada akhirnya fungsi ODS juga berkembang menjadi
semacam Desicision Support karena ODS mengandung
detail data yang telah terintegrasi dari berbagai sumber
Data.
(mengenai ODS, akan dibahas pada sesi tersendiri)

OLAP
Disebut juga sebagai Multidimensional Database (MDDB).
OLAP sengaja dirancang untuk memudahkan dan
mempercepat query yang dilakukan pada MDDB.

Karakteristik dan sifat OLAP sangat berbeda dengan OLTP
karena memang mempunyai tugas dan fungsi yang
berbeda.

ROLAP
Relational OLAP.

MOLAP
Multidimensional OLAP
(mengenai OLAP, ROLAP dan MOLAP, akan dibahas pada
sesi tersendiri)

End User Application
Sekumpulan Tool yang bisa melakukan Query, Analisa
Data, dan menampilkan informasi untuk mendukung
kepentingan Bisnis.

Paling tidak Tool ini juga dilengkapi dengan kemampuan
untuk akses Data, fungsi Spreatsheet, fungsi grafik, dan
ada fasilitas untuk menampilkan prompt dan
menyederhanakan tampilan screen pada End User.

End User Data Access Tool
Sebagai Client dari Data Warehouse. Pada Relational Data
Warehouse, Client dapat mengirimkan SQL Request
ke Server.

Secepatnya End User Data Access Tool akan selesai pada
SQL Session dan kembali untuk menampilkan pada screen
Data dari suatu Report, Grafik, atau yang lebih tinggi
seperti form suatu analisa pada User.

End User Data Access Tool dapat saja sederhana seperti
Ad-Hoc Query Tool atau bisa juga bisa komplex
seperti Data Mining yang canggih atau aplikasi modeling.

Ad-Hoc Query Tool
End User Data Access Tool yang dibuat khusus sehingga
memungkinkan User untuk membuat Query sendiri secara
langsung dengan cara memanipulasi Relational Tables dan
Join-join nya.

Dipasaran biasanya Ad-Hoc Query Tool dipaket menjadi
satu dengan Reporting Tool.

Modeling Application
Data Warehouse Client yang canggih dengan kemampuan
analytic yang dapat merubah bentuk atau menjadikan
intisari dari output Data Warehouse.

Modeling ini termasuk didalamnya adalah:
- Forecasting Model, yang berusaha memperkirakan masa
depan
- Behaviour Scoring Model, yang dapat mengklasifikasikan
perilaku belanja Customer
- Allocation Model, yang mengambil data Cost dari Data
Warehouse lalu membagikan Cost tersebut ke semua
poduct Group atau Customer Group.
- Terutama Data Mining Tools.

Meta Data
Semua informasi dalam lingkungan Data Warehouse namun
bukan data itu sendiri, Misalnya informasi mengenai
asal data source dari aplikasi apa, tabel apa, field apa,
formula bisnis bagaimana, dsb. Meta Data perlu dibuatkan
Catalog, diberi Version Stamp, di dokumentasikan dan di
backup.

Beberapa ETL Tool dipasaran ada yang sudah dilengkapi
Meta Data management.

Demikian elemen dasar dari Data Warehouse, masih banyak komponen yang nanti akan di jelaskan pada sesi berikutnya.

Reff:
Kimball, Ralph, 1998, The Data Warehouse Lifecycle Toolkit, Jhon Wiley & Sons, Inc.

OLAP Course Class 1

themarlin — Tue, 17 Nov 2009 18:36:49 +0000

November 12, 2009 · Leave a Comment · Edit This
OLAP Course Intro
Dimensions and facts: measures
Measure groups
ETL : fuzzy lookup transformation, logging execution results, placing checkpoints.
Define cube storage options.
KPI: orange,yellow,red.
Process and deploy a cube.
Why send a mailer selling a van to single people?
Most important: data-mining, prediction, if i follow this trend where i will be? SQL Server comes with 7 algorythms.
Use report manager to deply reports.

VVVVVVVVVVVVVVVVVVVVVVVVVVVV
Follow-up:
MDX to pull data from cube? Using SSRS? A:Yes. Only MDX pulls from cube.
Can storage be changed from MOLAP to HOLAP to ROLAP? (So some OLAP cube data is left in RDMS?) A: Dont have to rebuild, but may need to restart or redeploy.
Cubes are “read-only”, but sometimes users can writeback to update the cube? A: 99% of time read-only. Must spec in in build of cube you want writeback and all writebacks go to one table.
Datamart = cube, or a type of data in a group of cubes?
“Deploy” a cube?
“Deadlocks”? Long locks hold up the database, but deadlocks don’t let the table go free for other writes. Possibly due to two tables referencing each other. SQL Profiler helps show deadlocked processes….bugs in sql code.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Cubes are denormalized to speed up processing – as decision re what to include has already narrowed the bulk down.

Download SQL2008:
http://www.mhprofessional.com/downloads/products/0071549447/0071549447_Code.zip
Download databases:
See sqldownloadinstructions.txt in S: drive.
http://bellevuecollege.edu/lmc/media_reserve.html

What is BI? Method of storing and presenting key enterprise data to make decisions and forecast future results. Should we add more machines, or more sales people? Are we getting more requests or more customers? Less in some areas or for some products?

All the data must be converted into a cube periodically, so data in reports is not stale. Must collect data, and run ETL from one source into the cube. RDMS, Staging server, BI Cube (unified location of read-only data warehouse full of datamarts/cubes), Deploy reports to application server (LAN users) or a web server (WAN users). In some cases users can writeback to the BI server cube. IIS and web technologies should be separate from SQL server technologies.

XMLA: copying, backups, viewing metadata of your cube. Useful for giving admins metadata instructions to deploy cubes.

No locks needed on OLAP tables.

Snapshots, realtime, or cached queries.

Rapidly changing data (dollar amounts) vs slowly changing data.

OLAP Modelling
Schema: definition of data content and structure of tables.

Dimensions are the ways to slice your data, each possibly having multiple attributes. Do not look at relational tables to build dimension tables in your cube, look at the needs of user reports. Not all attributes need to be included.

FACT TABLE: aggregations of the facts/measures: units made, sales figures, and keys to dimension tables.

Star schema: one fact table, multiple dimension tables. Note that some cubes have two or more fact tables: resellersales and internetsales. (a not-so-strict-star schema) Tables are “flat”: no subcategories.

>>>>>>>>>>>non-relational cube tables duplicate data?

>>>>>>>>>>> none of the aggregations in a fact table are hierarchical groupings?

Snowflake schema: if goal is multiple drill down levels, hierarchical layers, then a snowflake schema is needed.

1. Grain statements: To being designing a cube, work from end to beginning, customer executive back to data needed. Start with “grain statements” : 1) What are key metrics, top interests, measures? 2) by what factors are key interests evaluated? by store, by area, by date? 3) by what level does each factor need to be evaluated? by store or city or region? by day or month?

With UDM, cubes can be built straight from the source, without a staging database.

>>>>>>>>>>>>>What determines the dimensions loaded by the client? “Significantly different from SSAS 2000″

2. How to maintain data, handle the changes, tweeking storage options: Update options are applicable to SCD and RCD’s. Must decide which dimensions are which, to determine how updates to cube are made.
>>>>>>>>>>>processing = ETL, aggregation, calculated measures?

Just the measures from a fact table (no keys) = the measure group, making report access easier as it will be in memory. multiple measure groups can be designed for a cube. eg,just the measures for 2005.

Action Modelling: Designing a trigger based on condition when building a cube that gives users the ability to drill-down on a cell.
Perspectives: Like views.
Translations: Gives the ability to pickup browser locale and give manually translated text for field names. Or set the format.

Categories: OLAP · Skills
Tagged: BI, CBI, OLAP, Skills

OLAP Course Class 1

themarlin — Thu, 12 Nov 2009 16:37:44 +0000

OLAP Course Intro
Dimensions and facts: measures
Measure groups
ETL : fuzzy lookup transformation, logging execution results, placing checkpoints.
Define cube storage options.
KPI: orange,yellow,red.
Process and deploy a cube.
Why send a mailer selling a van to single people?
Most important: data-mining, prediction, if i follow this trend where i will be? SQL Server comes with 7 algorythms.
Use report manager to deply reports.

Cubes are denormalized to speed up processing – as decision re what to include has already narrowed the bulk down.

XMLA: copying, backups, viewing metadata of your cube. Useful for giving admins metadata instructions to deploy cubes.

No locks needed on OLAP tables.

Snapshots, realtime, or cached queries.

Rapidly changing data (dollar amounts) vs slowly changing data.

OLAP Modelling
Schema: definition of data content and structure of tables.

FACT TABLE: aggregations of the facts/measures: units made, sales figures, and keys to dimension tables.

>>>>>>>>>>>non-relational cube tables duplicate data?

>>>>>>>>>>> none of the aggregations in a fact table are hierarchical groupings?

Snowflake schema: if goal is multiple drill down levels, hierarchical layers, then a snowflake schema is needed.

With UDM, cubes can be built straight from the source, without a staging database.

>>>>>>>>>>>>>What determines the dimensions loaded by the client? “Significantly different from SSAS 2000″

OLAP Course Class 2

themarlin — Thu, 12 Nov 2009 05:47:29 +0000

Create a new vs ssas project, and a db, both named “MaxMinSalesDM”. Use the Simple db recovery option.

Sometimes the Time dimension doesn’t need to be created on its own because other tables may already have time-related data in them.

Create a cube, choose ‘Generate a schema for every row’. Starts the wizard to help you define attributes for the dimension tables.
4 Items that create a cxn: Server,Database,Authentication,Provider (Oracle,SQLExpress,SQLServer,etc.)

Windows authentication will take the currently logged on person. Impersonation will always use the set user account and pwd instead of the logged on user. 4 Impersonation options: ‘SQLServer authentication’ needs to be allowed on the sql server, and an account needs to be created.
“Use the Service account” is NTAuthoritySystem which has higher privileges than a guest. VVVVV Need sql admin to allow this, or creating cube configures this? “Use the credentials of the current user” will work when the Ops or SQL Admins have given you access. “Inherit” permissions of “caller” VVVV

>>>>A view is a saved Select statement?
Design and Build a Cube:
1.Create Dim and Fact tables. 2.ETL fill tables 3. Design the cube: Calcs,KPI’s,Actions: Write the statements defining the code 4.Process (compile and do calculations) and deploy the cube to a SSAS cube.

Snowflake schema has hierarchies.

SSIS:
Control Flow is made of ‘buckets of tasks’, prebuilt or custom built, some having data flow actions and others run a script, and others ftp some files or email someone.
Connection Managers define the interfacing of the package.

“External Column” is fieldname in source, “Output Column” is the … as, what you might want to name it.

Red output arrow is ‘what to do if this task throws an error’.
Employee$ is what? vs Employee ?

>>>>>>A “named query”?
>>>>>>create and consume flat files into/from where

A split transform: Default output is all data outside conditions, the leftovers that you go to a different destination.
Make a habit of rendering the Mappings before filling a Preview.

Degenerate Dimensions.

nirupampratap — Mon, 19 Oct 2009 06:55:53 +0000

I was going through the design of one of our OLAP databases and in that database I found certain fact tables with columns which seemed to be fitting the description of a dimension rather than that of a fact/measure. I had long arguments with the designer who designed this database and still couldn’t get a convincing answer. He seemed to have started designing the database assuming this to be a fact whereas now he was confused between terming that field as a fact or a dimension. So, I did some reading on this aspect and found this article describing something called as ‘Degenerate Dimensions’. It explained in detail why some columns are more like dimensions and how they need to be modeled.

As such this discussion happened between us some months ago… but I started loosing track of the URL… hence I thought it might be a good idea to track it as part of a blog post . Also, it might be of some use to people who are designing OLAP applications.

Pratap.

dbta.com: New Additions to the DBA Bookshelf

Visitor Blogs — Fri, 16 Oct 2009 02:49:48 +0000

SQL Injection Attacks and Defense by Justin Clarke.

Microsoft SQL Server 2008 Reporting Services Unleashed (SAMS Publishing).

Oracle Essbase 9 Implementation Guide (Packt).

Kevvie Fowler’s comprehensive tome, SQL Server Forensic Analysis (Addison Wesley). www.applicationforensics.com.

Fowler’s other Web site is Ring Zero (www.ringzero.ca)

Chapter 1. Introduction to Databases

Running Chapter 1 Sample Scripts

Databases Explained

How Databases Are Used

Databases and COTS Applications

Database Structure

Structured Query Language (SQL)

Database Transactions

The ACID Model

Referential Integrity

Summary

Chapter 2. SQL Server Fundamentals

History of SQL Server

SQL Server Versions and Editions

Architecture

SQL Server Connections

Dynamic Management and Database Console Commands

Logging

SQL Server Agent

Summary

Chapter 3. SQL Server Forensics

The Road to SQL Server Forensics

SQL Server Forensics

SQL Server Forensic Methodology

Summary

Chapter 4. SQL Server Artifacts

SQL Server Artifacts

Resident SQL Server Artifacts

Nonresident SQL Server Artifacts

Artifact Summary

Summary

Chapter 5. SQL Server Investigation Preparedness

SQL Server Investigation Preparedness Overview

Configuring Your Forensics Workstation for a SQL Server Investigation

Creating a SQL Server Forensics Incident Response Toolkit

Summary

Chapter 6. Incident Verification

Running Chapter 6 Sample Scripts

Incident Verification Explained

What Not to Do When Investigating a Live SQL Server

Responding to an Incident

Identifying the SQL Server Instance Name

Connecting to a Victim System

Disconnecting from the Victim System

Identifying Signs of an Intrusion

Submitting Preliminary Findings

Summary

Chapter 7. Artifact Collection

Focus on Ad Hoc Collection

Running the Sample Scripts

Maintaining the Integrity of Collected Data

Automated Artifact Collection via Windows Forensic Toolchest

Identifying the Victim’s SQL Server Version

Ad Hoc Artifact Collection

Collecting Volatile SQL Server Artifacts

Collecting Nonvolatile SQL Server Artifacts

Summary

Chapter 8. Artifact Analysis I

Working Along with Chapter 8 Examples

Pre-analysis Activities

Authentication and Authorization

Configuration and Versioning

Summary

Chapter 9. Artifact Analysis II

Working Along with Chapter 9 Examples

Pre-analysis Activities

Activity Reconstruction

Data Recovery

Summary

Chapter 10. SQL Server Rootkits

Traditional Rootkits

SQL Server Rootkits: The New Threat

Generations of SQL Server Rootkits

First-Generation SQL Server Rootkits

How Rootkits Can Affect a SQL Server Investigation

Detecting Database Rootkits

When to Check for Database Rootkits

What to Do if You Find a Rootkit

Summary

Chapter 11. SQL Server Forensic Investigation Scenario

Scenario Overview

Importing Sample Artifacts

Investigation Synopsis

Incident Verification

Artifact Collection

Artifact Analysis

Activity Reconstruction

Investigation Summary

Appendix A. Installing SQL Server 2005 Express Edition with Advanced Services on Windows

Appendix B. SQL Server Incident Response Scripts

SSFA_DataCache.sql

SSFA_ClockHands.sql

SSFA_PlanCache.sql

SSFA_RecentStatements.sql

SSFA_Configurations.sql

dbta.com: New Additions to the DBA Bookshelf.

Nigel Pendse Sharply Criticises Traditional BI Providers at the 2009 Palo Open

Andy Painter — Thu, 15 Oct 2009 08:59:43 +0000

OLAP Guru Nigel Pendse Recommends Smaller Software Providers

In his speech, keynote speaker Nigel Pendse, software analyst and publisher of BI Survey, sharply criticised large producers of Business Intelligence software. The annual meeting of users and partners of the open source software Palo was held last Tuesday in Frankfurt, where applications and solutions that use Palo were introduced, and the 2009 Palo Award was conferred.

Based on the feedback from Business Intelligence users, Pendse demonstrated that BI products and services of large providers performed well below average. Pendse explained that Business Intelligence is simply not the core service of large software companies. Large providers’ ongoing acquisition politics reflects the poor quality of their product portfolio. According to Pendse, nothing positive can currently be expected from the large providers in terms of BI, since they are overly concerned with integrating their many new acquisitions. Pendse advised corporate BI users to choose the best products on the market, which generally are offered by smaller providers such as Jedox.

Finance applications were the focus of the presented Palo applications. The auditing and consulting firm PriceWaterhouseCoopers AG introduced its new Palo-based applications in this area under the slogan ‘Treasury Intelligence’. These include solutions for monitoring the currency risks, credit risks and counterparty risks, liquidity planning and creation of IFRS 7-conformant appendix information. Thomas Hampel, a senior consultant at PriceWaterhouseCoopers, identified speed and user-friendliness as the advantages of the Jedox technology, qualities that also make it possible to easily adapt the product to the needs of individual clients. The main advantage identified by Hampel was that Palo ‘brings Excel applications to the Web at the push of a button, so to speak.’

The software company elkomSolutions introduced a Palo application for consolidated financial reporting that provides consolidated financial statements in accordance with HGB, IFRS or US GAAP standards. Insurer Standard Life presented its in-house budget planning and reporting system, which is based on the Palo suite. Ingo Weishaupt, Head of Management Information at Standard Life Group Germany had this to say about the Jedox solution: “A successful software project, implemented on time, it paid for itself in the first year and has been running reliably for three years.”

Oracle Database 11g vs. Microsoft SQL Server 2008

Visitor Blogs — Sat, 10 Oct 2009 03:12:16 +0000

This report is the latest in a series of Comparative Management Cost Studies (CMCS) comparing Oracle Database with database management systems offered by other leading enterprise software vendors. This edition of the study compares Microsoft SQL Server 2008 with Oracle Database 11g.

Following the same approach taken in the earlier studies, Edison Group set up a laboratory environment to analyze a suite of standard RDBMS administrative tasks, and measured their respective management efficiency (time taken to complete tasks) and their complexity based on a proprietary manageability metric. Using the management efficiency results, Edison Group calculated the annual savings for businesses due to the enhanced DBA productivity that would result from using the product with superior manageability.

The study results show that, while both the vendor products have features to help average database administrators (DBA) perform everyday administrative tasks, Oracle Database 11g holds a substantial advantage over Microsoft SQL Server 2008. Our detailed analysis reveals that:

 DBAs can perform typical administrative functions in 41 percent less time when using Oracle Database 11g compared to Microsoft SQL Server 2008.
 Oracle Database 11g requires 43 percent fewer steps for the same set of standard RDBMS tasks than Microsoft SQL Server 2008 using Edison’s metric for complexity assessment.
 Benefiting from increased DBA productivity due to lower complexity and higher efficiency cited above, businesses could save up to $33,520.47 per year per DBA by using Oracle Database 11g rather than Microsoft SQL Server 2008.
The main areas of difference we discovered between the two products were:
 Backup & Recovery: In the case of backup and recovery tasks, Oracle Database 11g offers architectural and functional capabilities beyond those offered by SQL Server 2008. Oracle Database 11g took 53 percent less time and 60 percent fewer steps than Microsoft SQL Server in backup and recovery tasks.
 Performance Diagnostics and Tuning: In this category, Oracle Database 11g demonstrated a significant 87 percent savings in time.

In the course of this study, Oracle Database 11g and Microsoft SQL Server 2008 were compared against a set of methodology metrics in order to determine which of the two products is easier to operate for businesses with real-world database management requirements. The Test Administration Workload Task Areas that we used to perform this study fall into the following four categories:
 Database Installation and Setup
 Day-to-Day Database Administration
 Backup and Recovery
 Performance Diagnostics and Tuning

Task categories were divided into individual tasks that logically map into their respective areas. To determine the overall manageability of a given task for a given product, each task was broken down into steps to assess the complexity and usability involved. Each task was then measured for time and number of steps required for successful completion.

Next, tasks were weighted against workload weighting constants. These weightings were used to determine the relative importance of a given task as measured against all of the tasks required to manage the entire product administration lifecycle. In other words, simple tasks that occur relatively infrequently were given a proportionately lower weighting than complex tasks that occur on a regular basis.

Finally, the results were tallied and the CMCS metrics for each product were substituted into manageability cost formulas to determine the projected human resources cost of operating both products, based on median DBA salary.

The Methodology Defined

For purposes of this study, the methodology is defined as a product manageability cost evaluation process, whereby the two products in question are compared against a set of task-oriented objective and subjective metrics in order to derive an accurate set of analytical results. The outcome of this study determines the Comparative Management Cost (CMC) incurred by managing and operating either of these products in a production environment. The methodology employed to conduct this comparison consists of the following elements.

Workload Weighting

The workload weighting is a set of constants that define the relative importance of a single task area in the workload, based on frequency of execution and measured against the entire set of task areas that compose this study.

The Study

The study is the baseline checklist of standard database administration tasks routinely performed, which are quantitatively and qualitatively compared in order to objectively determine, on a task-by-task basis, which product is superior. This is measured primarily in terms of ease of administration and secondarily (for certain tasks only) in terms of system speed of execution — the wall clock time it takes for the system in question to complete a job once it has been submitted by a DBA. The function of this study is to apply a set of quantitative metrics, developed by Edison Group, to a list of tasks typically regarded as qualitative in nature, in order to derive a meaningful set of CMCS statistics that can reveal the real difference in management costs for the two products in question.

Tasks

A task is defined as a complete logical activity, composed of one or more steps, all of which effect a significant alteration on the state of the database that accomplishes a specific work goal. Each task is measured for time and complexity. Time and complexity, as measured in the study, are defined as follows:

Time

Defined as the amount of time it takes to perform a given task. For certain (asynchronous) tasks, when a job can be run in the background so that the DBA can use the time for accomplishing other tasks, time is measured strictly in terms of the time it takes the DBA to perform the steps to configure, initiate, and submit a given task.

For other (synchronous) tasks in the study that demand the DBA’s full attention and prevent the accomplishment of other tasks (as in performing a hot recovery operation on a live database), time is measured to include both the time it takes for a DBA to configure/execute the task in question as well as the time it takes the system to complete the task. All time metrics are measured in wall clock time.

Complexity

For the purposes of this study, complexity is measured using a proprietary metric devised by Edison Group. It is defined as the number of system-affecting steps it takes to complete a given task, where a step is defined as a task component that effects a change of state to the database.

Because not all steps have the same inherent complexity, each step is further broken down into increments to account for the difference. An increment is a decision point that the user must make to complete a step. Increments are technically defined as a part of a step that will have a measurable effect on the state or execution path of that step in the task process, but which in and of itself does not affect a change upon the underlying database state until the step being executed is complete. For example, selecting Basic vs. Advanced Install with the installation wizard is an increment and not a step.

Complexity is then measured in terms of number of steps, but taking into account the following factors:
 The number of increments it takes to complete each step.
 Whether or not instrumentation for a given step is GUI-based or requires the use of a command line/scripting interface.
 Whether or not the task requires a context switch between multiple interfaces in order to be completed. If a context switch exists, then additional steps will be added to the total step count for a given task.
 The above factors affect the complexity calculation as follows:
 The primary measure is steps. If a step has many increments, it is considered several steps. The metric allows each step five increments, and thereafter we add steps for each additional five increments rounded up.

So if a step has between 0–5 increments, it remains unchanged; if it has between 6–10 increments, it is increased by one; between 11–15 increments, it is increased by two; and so on. We decided to do this because, while increments are secondary to steps in determining complexity, they do modify the relative complexity of a given step in the course of completing a task. In other words, steps with a low number of increments are simple, and steps with a high number of increments are complex.

 The other modifiers (instrumentation and context switching) occur very infrequently in the products under review, but they were significant enough a factor that we needed to account for them in some meaningful way in order to generate a measure of complexity that accurately reflects our experience of using the two products.

 Regarding instrumentation, if an operation could be executed entirely within a GUI interface, then the complexity/step value for that task would remain unmodified. If, on the other hand, a step required the use of a command line interface, this would increase the step count. For a simple single-line command operation, the step count was increased by one, whereas if the operation required the user to write a script, the step value was increased by two or more, depending on how much work was required to write the script in question.

Lastly, we come to the matter of context switching. If a context switch was encountered during the course of completing a given task, then two or more steps were added to the step count for that task. The possible addition of more than two steps was allowed for as a judgment call on the part of the analyst performing the task under consideration. The reason tasks containing context switches were penalized is that we regard as inherently more complex to understand the dependencies of relating and performing a single operation in two different environments in order to complete a single task, as opposed to performing a similarly complex task in a well-integrated environment where all the operations can be accomplished in one place.

The workload for this CMCS was reduced to the basic set of atomic maintenance operations that effectively fulfill all fundamental database administration procedures. The reasoning behind this approach is that enterprise-class database configuration and administration is a non-trivial matter; we therefore set out to develop a (relatively) simple yet comprehensive evaluation process, establishing a CMCS methodology benchmark that we feel is realistic in its technical assessment, yet accessible to the large audience of non-technical decision makers who will read this document.

The evaluation of each task in the study workload was executed by measuring the time that a typical DBA spends in a given workload task area using the methodology metrics. We counted compute time for utility tasks only in situations where the DBA is expected to “baby sit,” such as instance recovery.

This was the process we used in determining whether Oracle Database 11g is less expensive to operate than Microsoft SQL Server 2008.

Test Administration Workload

Installation and Simple “Out-of-Box” Setup Tasks
 Install db/software/out-of-box setup
 Create second database server/instance
 Setup proactive monitoring

Day-to-Day Database Administration Tasks

 Create user with roles, privileges
 Create tablespace/filegroup
 Add space to database
 Create table
 Create index
 Reclaim space due to fragmented data

 Load data from text file
 Adaptive thresholds and workload comparisons

Backup & Recovery Tasks

 Configure and perform full backup
 Recover dropped table
 Recover data file
 Recover from erroneous transaction
 Recover from multiple failures

Performance Diagnostics & Tuning Tasks

 Diagnose performance problem
 Tune resource-intensive SQL
 Tune memory

Task Areas/Tasks

The following glossary will be updated as required to reflect changes in the products and tasks performed.

Database setup and configuration (workload task area): A workload task area that encompasses all of the operations that a DBA would need to perform in order to accurately install and configure either product prior to using it in a real-world application.

Install db/software/out-of-box setup: The workload task that identifies the process of installing and configuring either of these products for the first time.

Create new database server/instance: The workload task that allows a DBA to create a second database server/instance on a computer already running one or more instances of the RDBMS.

Set up proactive monitoring: Proactive monitoring is a tool that allows a DBA to identify problems with the RDBMS before they become performance or operational issues. Setting up monitoring is crucial in providing a highly reliable system.

Day-to-day database administration (workload task area): The workload task area where all of the routine DBA operations occur, such as creating database users and objects (for example, tables, indexes, triggers, procedures), as well as granting and revoking roles/privileges, plus database table/tablespace/data file sizing.

Create index: The workload task that allows DBAs to create an index on a set of columns in a table that are heavily queried in order to speed the execution of queries run against that table.

Create index on partitioned table: The workload task that allows DBAs to create local indexes on the partitioned table to help speed the execution of queries run against the table.

Create Partitioned Table: The workload task that allows DBAs to create a partitioned table object to store information inside a schema in the database.

Create user with roles, privileges: The workload task that allows DBAs to manage user security in the database.

Create schema: The workload task that allows DBAs to create a new schema.

Create table: The workload task that allows DBAs to create a table object to store information inside a schema in the database.

Reclaim wasted space from tables with fragmented data: The workload task that allows DBAs to pack/shrink the database after prolonged use in order to consolidate space and optimize performance.

Load data from text file: The workload task that allows a DBA to load information from an external source such as a flat file or spreadsheet into one or more tables in the database.

Create tablespace: The workload task that allows DBAs to build a new tablespace for use by a particular database instance or schema.

Add space to tablespace: The workload task that allows a DBA to add data files to a tablespace, thus increasing the amount of space available to all of the objects contained in that database.

Backup and recovery tasks (workload task area): The workload task area where all tasks pertaining to database backup and recovery are performed.

Configure and perform full backup: The workload task that allows DBAs to schedule and execute regular system backups as part and parcel of standard system fault tolerance operations.

Recover dropped table: The workload task that allows a DBA to recover a table that has been inadvertently dropped from the database by a DBA, developer, or power user in the course of working with the database.

Recover data file: The workload task that allows the DBA to recover a data file from a backup copy in the event of a media failure.

Recover from erroneous transaction: The workload task that allows a DBA to undo a mistakenly executed transaction in order to recover objects ill-affected by this transaction to the state they were in before the transaction was executed.

Performance diagnostics and tuning tasks (workload task area): The workload task area where all performance-related diagnostic, tuning, and optimization tasks are performed.

Diagnose performance problem: The workload task that allows a DBA to analyze a poorly performing system in order to assess the performance problem as a prerequisite to performing the systems optimization functions required to bring the level of system performance to an optimal state (such as creating additional indexes, tuning a query/procedure, defragmenting tablespaces, or adjusting the server’s memory configuration).

Tune instance memory: The workload task that allows DBAs to adjust server/instance system memory configuration parameters in order to efficiently support the load placed on the system by all applications that access the database.

Fix performance problem (tune SQL statement): The workload task that allows DBAs to optimize a poorly running query/procedure so that it executes in an efficient manner.

Chapter 2: What Should You Expect from Your Data Warehouse?

Visitor Blogs — Fri, 09 Oct 2009 19:12:46 +0000

ISBN: 978-0-470-40747-9
Copyright of Wiley Publishing, Inc.
Indianapolis, Indiana

In This Chapter
▶ Making business decisions based on facts, not intuition
▶ Realizing the value of “data at your fingertips”
▶ Looking at cross-organization communications and your data warehouse
▶ Changing your business because of insights from data warehouse information

All too often, the members of a data warehouse development team
proudly unveil their creation, get a few “oohs” and “aahs” from their
user community, and then find out that “if you build it, they won’t necessarily
come.” The data warehouse just sits there, quietly being restocked every
week or every month, and supporting only a few random user information
requests until plans eventually are drafted for its replacement.

You can avoid many, or perhaps most, of these unfortunate situations if
everyone, from the executive sponsors to the technicians and developers,
focuses their efforts on a single question: “What do you do with a data
warehouse?”

The contents of the warehouse (the data) aren’t important; rather, how
the data is used in everyday business life is what makes the warehouse
really useful.

Using the Data Warehouse to Make Better Business Decisions

Suppose that John is a district manager at MegaRetroMania, Ltd., a national
chain of video-rental stores that specializes in movies from the 1950s and
earlier. Mary is John’s counterpart in another district. Both excitedly awaited
26 Part I: The Data Warehouse: Home for Your Data Assets
the rollout of MegaRetroMania’s brand-new data warehouse, an event that
finally occurred a month ago. John and Mary both attended the data warehousing
orientation class in which they figured out how to run reports and
make information requests by using the contents of the warehouse. Each
person received a handsome, ready-for-framing certificate indicating
successful completion of the course.
On July 20, the day after June’s results had been finalized and loaded into the
data warehouse, John sits down at his PC, clicks the icon that enables him to
access the data warehouse, and makes this request:
Show me, for all my stores, a breakdown of second-quarter sales compared
to first-quarter sales, each store’s second-quarter sales from a year
earlier, and the sales of all competitors within two square miles of each
store’s location.

While John watches the little “Working, please wait” hourglass icon twirl for a
few seconds on-screen, he thinks, “It’s fascinating how they can make it seem
like the sand is really sifting from top to bottom like that.” After the results of
his request are displayed on-screen, he prints the report, walks over to the
printer to retrieve the five printed pages, flips through them for a few
moments, and then thinks, “That’s really neat!”

Then he slips the report into the middle drawer on the left side of his desk
before leaving for lunch.

And he never looks at it again.

Mary runs the same identical report for her district; after printing the report
and flipping through the pages, however, she (unlike John) has a pen in hand
and circles one number, underlines another, and writes “Uh-oh!” next to the
line containing information for a store that she knows is having problems.
Then, Mary returns to her desk, report in hand, and picks up the telephone.

First, she calls her problem-child store and, after the manager comes to the
phone, spends the next half hour making follow-on information requests
from the data warehouse, discussing the result of each request with the store
manager, and considering various options, such as sales promotions.

After Mary takes her regular two-mile lunchtime walk, she calls the manager of another store in Madison County, Iowa, whose second-quarter sales were relatively flat after experiencing at least 15-percent growth every quarter for the past three years (a fact she obtained from the data warehouse when she noted the lack of growth in the second quarter). While the store manager is on the phone, Mary gets a week-by-week sales breakdown from the warehouse, which shows that the first three weeks of the quarter had been fairly good, with sales during each week slightly ahead of the preceding one.

During the fourth week of April, however, a large dip occurred, followed by
an even larger dip the following week and yet another drop the week after
that (the worst week any store had ever had, it turns out, in the long,
distinguished history of MegaRetroMania) before sales began picking up
in mid-May.

Suddenly, the store manager has an idea — maybe the Olympic Games were
causing the problem! The Olympics had been held in Madison County for
the first time ever, and everyone had attended. Even that store’s primary
customer base, the legions of photographers who usually come to Madison
County to take pictures of covered bridges during a four-day period and rent
movies to watch in their motel rooms at night, were preoccupied for those
three weeks by the Olympics.

Satisfied that no burgeoning problem is waiting in the Madison County
store — just a once-in-a-lifetime anomaly, most likely — Mary turns her
attention to the other annotations on her printout. By the end of the day,
five stores have action plans in place for special promotional campaigns
to combat competition that’s opened in the past few months near those
MegaRetroMania locations.

Here’s a quiz. True or false: Mary and John both used the data warehouse
today.

The answer: False.

You might be thinking, “What? Of course John used the data warehouse
today. He ran a report and looked it over.”

Although John accessed the data warehouse, he didn’t use it. He looked over
a printout and stuffed it away forever, and then returned to business as
usual. When you compare his action to Mary’s, the differences are obvious.
No matter what else appears on a data warehousing project’s mission
statement, and no matter what the project’s sponsors say to convey the
project’s merits to the people who control funding, the primary purpose
of a data warehouse is to help people make better business decisions. Data
warehousing isn’t about simply accessing data and then doing nothing with
it; it’s about really using data.

Here’s one way to make sure that data warehouse users act more like Mary
than John: When you conduct training sessions before you turn users loose on
the new data warehouse, explain to them not only how to access data (which
types of queries and reports they can run or which types of data are available,
for example), but also include real-world examples of how to use the results of
their warehouse access.

The training session for MegaRetroMania district managers (and other warehouse
users), for example, might feature end-to-end hands-on training that
explains how to get sales reports and what the reports look like. To make
sure that users understand the usefulness of the data warehouse, the instruction
could also feature role-playing or some other type of training in what to
do with those reports to improve the stores’ performances.

Finding Data at Your Fingertips

Everyone has heard the phrase “information (or knowledge) is power,” and
the more skillfully people use information in the course of their jobs, the
greater their chances for success.

The process of gathering data from many different sources (if you can
even get the data at all) has traditionally been tedious, particularly before
computers became commonplace — and it has remained that way during
the information age. Some novels and movies would have you believe that
after you press just a few keys, you can automatically access vast amounts
of data from anywhere in the world, regardless of the platform you’re using,
the structure of the data (how it’s organized), or how the data is encoded
or keyed.

The real world isn’t quite that orderly, and anyone who has struggled to
perform what should be simple tasks (merging Pacific Rim sales data from
one system with North American sales data from another, for example) is
probably aware of the difficulties.

Suppose that Steve is a district sales manager at BlackAndWhiteVideos, Inc.,
MegaRetroMania’s upstart archrival. Unlike John and Mary, though, Steve has
no data warehouse from which to make requests, such as “Show me the top
20 BlackAndWhiteVideo stores across the district in which monthly total
comedy video rentals are at least double those of action videos so that I can
adjust the inventory accordingly.” Instead, Steve has to get the answer to
this question the old-fashioned way, as shown in Figure 2-1, by following
these steps:

1. He writes and distributes to all store managers in the district an e-mail
stating that he wants a report sent to him no later than the fifth day of
each month of rental dollars by category (such as comedy, action, and
romances), based on transaction records from each store’s point-of-sale
(POS) computer system.
2. On the fifth day of each month, he looks through the reports received
and notes each store from which he hasn’t received a report.

3. He spends most of the next two days calling store managers who haven’t
sent reports and hearing something like, “Oh, wow, I totally forgot all
about that; what do you want me to send you again?”

4. While waiting for each of the missing reports, he begins entering into a
spreadsheet the numbers from the reports received so far.

Each month, it occurs to Steve that he should send a new e-mail asking
managers to e-mail him a disk with each store’s data on it (already in
spreadsheet format) so that he doesn’t have to reenter the information.

He never quite gets around to that request because he assumes that it
takes several months to train someone in each store to be able to
extract data from the POS application into a spreadsheet.

5. By the 12th of each month, he makes another round of phone calls to
the stores that still haven’t submitted their reports, and this time
threatens to — well, you get the idea.

6. Finally, by the 20th of each month, after the spreadsheet program has all
the necessary information, he runs a few simple sorting routines to get
the necessary reports.

7. And he prepares to start all over for the next month’s data.

I have four questions about Mary at MegaRetroMania and Steve at
BlackAndWhiteVideos (forget about John at MegaRetroMania — he’s a loser):

✓ Who’s making more effective use of technology?
✓ Who’s likely to be more productive in a district manager’s role?
✓ Who’ll likely make more timely business decisions with (using businessschool talk) a more positive impact on short-term and long-term revenues (top line) and profit (bottom line)?
✓ Who’s more likely to be frustrated and look for another job that involves much less wasted time?

(Answers: Mary, Mary, Mary, Steve)

Both Mary and Steve could have access to the same information and in the
same way because both MegaRetroMania and BlackAndWhiteVideos have
similar computer systems with similar data. The difference is that because
MegaRetroMania has invested in data warehousing technology, Mary and her
counterparts (even that loser John, if he would only get with the program)
can make information-based and timely business decisions.

Here are two more questions to consider:

✓ Does your current job more closely resemble Steve’s or Mary’s in how you’re able to get access to information you need?
✓ Based on how you answered the first question, would you rather have a job more like the other’s?

Facilitating Communications with Data Warehousing

A benefit of data warehousing that’s much less tangible than having information
for better business decisions is that data warehousing often facilitates
better communications across a company than what existed before the
warehouse project began:
✓ The information technology (IT) organization — the organization that handles infrastructure (hardware and software platforms, networking, and communications, for example) — begins working more cooperatively with its customers in the business organizations.

✓ Business organizations that are drawn together as part of a data warehousing project often gain more appreciation for each other’s missions and challenges.

Part of the better communications picture occurs (usually) between applications development and customers in the organization’s business units.
Data warehousing can promote increased levels of communication across different business units.

IT-to-business organization communications

If you’re a veteran of an IT-organization-versus-business-organization war in
your company, you might have won some battles and lost others. Make no
mistake about it, though: The other side is the enemy!

If you’re on the IT side, you might grumble about these issues:

✓ Business users make unrealistic requests for new applications and enhancements to existing applications.
✓ Users don’t participate during application development and testing.
✓ Users are so technology-challenged that they require hand-holding for the simplest application functions.

On the other side, business users commonly make these types of complaints:
✓ IT application developers have no idea about how the company’s business is run and usually don’t even bother to ask.
✓ Developers just put whatever features they want into the applications, regardless of whether those features are applicable to the business processes.
✓ IT is too slow in responding to requests for support.

Since the early 1980s, when the then-new personal computer and local-area network permitted business organizations to take direct responsibility for part of their information management and application capabilities, many companies have experienced an era of mistrust and lack of communications between IT and business. This level of mistrust has continued to grow with the proliferation of business productivity tools such as Microsoft Excel — enabling users to formulate their own solutions without IT. And the war within IT has also grown with the concept of Central IT, which manages the shared services within IT, and Line of Business (LOB) IT, which is closer to the business customer and manages his or her specific applications. Often, the data warehouse is managed by Central IT, which isn’t allowed to talk with the business — people from Central IT can talk only to the LOB IT personnel.

Talk about dysfunctional!

Data warehousing projects often give organizations a chance to mend the rift
between IT and business users, as well as LOB IT and Central IT. The nature
of data warehousing (for example, the focus on business questions and the
data necessary to answer those questions) gives people from both sides an
opportunity to better understand the other group’s roles and concerns.

Increased IT-business organization communications often leads to more
effective and efficient working relationships because everyone develops
a degree of trust. A successful project or two usually can do wonders in
achieving increased cooperation.

You don’t necessarily get better communications between IT and business
organizations just because they jointly embark on a data warehousing effort.

For example, if you use the word “dysfunctional” to describe the relationship
between IT and the business community in your company, you might want
to look at some team-building workshops or other remedial training before
proceeding with a data warehousing project.

Communications across business organizations

Data warehousing involves, in many ways, the use of technology to break
down barriers that are inherent in large companies, such as different computer
systems performing similar functions for different groups of users, or
different databases with information that would be better kept in a single
database. A sales-and-marketing organization responsible for one of its
company’s products naturally wants to help integrate its data (performance
results) with that of another sales-and-marketing organization, responsible
for a different product, that uses a different computer system (maybe
because of a corporate acquisition), right?

Corporate intrigue and turf wars are the hallmark of business. Business
executives and their staff members usually think of information not as a
corporate strategic asset, but rather as a personal one.

The breaking-through-barriers nature of data warehousing usually takes a
top-level mandate from a chief executive officer (CEO) or chief operating
officer (COO): “Thou shalt all cooperate, or thou shalt look for another job.”
That type of mandate is probably a corporation’s best chance for overcoming
detrimental corporate politics.

Suppose that you’re a senior company executive at a consumer products
company, and that company has an environment that looks like the one
shown in Figure 2-2. You decide that you want to dedicate a large part of your
capital budget to enable your managers and analysts to get timely access to
data so that they can make better business decisions. Implementing a data
warehouse would move the data from the separate division’s product sales
applications into one central location for the users to access.

Using the cooperate-or-else mandate, you can ask — and receive answers
to — the barrier-busting questions that present you with a consolidated
picture of your sales, regardless of how many source systems the data come
from. In addition, after the first successful collaborative, cross-organization
data warehousing effort, people probably feel less territorial and work more
closely together for the common good of the company.

Or they might not. Any lack of cooperation on the part of the leaders and
members of a particular business organization, however, becomes extremely
apparent because their data either aren’t in the warehouse (often called a
data gap) or, worse, the data warehouse project fails — and you can easily
tell who’s at fault, the data gap owner.

I don’t mean to imply that you need to make finger-pointing and blamecasting
a part of your data warehousing project; rather, you should implement,
from the beginning, a clear directive for cooperating with peers that
everyone has to follow.

Facilitating Business Change with Data Warehousing

Data warehousing involves facilitating change in business processes. In
addition to being able to make better, information-driven operational and
tactical decisions, you gain insight into key areas that can help you make
strategic decisions about the fundamental aspects of your business.

Your data warehouse can act as an early-warning system to let you know
that you might need to make some major business changes. Imagine that
it’s 2000 and you just got into the video-rental business. You’re the part
owner and purchasing manager for a regional chain of retail stores that rents
digital video discs (DVDs) and video games, and sells compact discs (CDs). If
you had a data warehouse that gathered information from the point-of-sale
computers in all your stores and external data about your competitors’ sales
activity (purchased from an external source), you probably would have
noticed a couple of interesting trends when you looked at product revenues
over time:
✓ Sales of music in their historically popular CD format have dropped off remarkably. Because your competitors’ sales for those items are also way off, you might decide that you don’t need to devote the shelf space
to stock these types of items anymore.
✓ Video games for Playstation 2 (PS2) are rented much more frequently than the same games in Dreamcast format. Because the video game business is rapidly evolving (remember, it’s 2000), your rental numbers
might be declaring PS2 the winner of the next-generation gaming consoles and that the Dreamcast games are another item probably not worth ordering and putting on the shelves.

Both these decisions are, for the most part, tactical because they involve
product-stocking decisions. Neither decision fundamentally changes the way
your company does business, but rather helps you decide which products
you want to keep in stock.

But suppose that trend-line information coming back from the data warehouse
shows relatively flat sales of all types of music and dramatically
increasing revenues from DVD and video game rentals. The future of your
chain might lie not in sales of music at all, but rather in DVD and video game
rentals. Rather than have to rely on a gut feeling, you can make data-driven
strategic decisions about how your business should change.

Imagine having to make the same type of decision in 2008 if you’re in the
automotive business. Gas guzzler sales are plummeting as fast as the price of
gas is increasing. While worldwide demand for oil grows, the supply and
demand curve isn’t friendly to traditional car sales. The highly positive
growth curve that sport utility vehicles (SUVs) had a few years ago is giving
way to the hybrid and green products. Again, your data warehouse gives you
the information to make data-based decisions, rather than have to rely on
hunches or, at best, bits and pieces of data gathered from various computer
systems.

Technical Session de Business Intelligence por Ismael Soares

andrefaria — Thu, 08 Oct 2009 16:33:59 +0000

Na Technical session desta semana Ismael Soares introduziu conceitos de Business Intelligence como Data Warehouses, Data Mining, Data Marts, Cubos OLAP e muito mais.

Veja os slides no slideshare e asssita a palestra na integra em nosso canal do vimeo:

Ismael Soares é desenvolvedor Java EE na Bluesoft, estudante de banco de dados na Faculdade Impacta de Tecnologia. Tem mais de sete anos de experiências em análise e desenvolvimento de software. Já trabalhou com diversas tecnologias entre elas: VB6, ASP, PHP, C#.NET, VB.NET, ASP.NET, PL/SQL e T-SQL. Já participou de diversos projetos de banco de dados entre eles: Oracle, MSSQL Server, MySQL e Postgree.

RapidMiner -- Data Mining, ETL, OLAP, BI...

sumodirjo — Fri, 02 Oct 2009 23:36:13 +0000

RapidMiner — Data Mining, ETL, OLAP, BI

ETL, data warehousing, data mining, OLAP, business intelligence (BI) in Java. 500+ modules: extract, transform, load (ETL), data mining, data analysis + Weka, statistical forecasting, preprocessing, validation, visualization, OLAP, business intelligence.

Download

Supported By : rapid-i.com

SAP BI - part 1

bsoft07 — Fri, 02 Oct 2009 16:16:24 +0000

Salam BSoft… Akhir-akhir ini saya sedang tertarik mempelajari SAP BI (Bussiness Intelligence).

OLAP models and OLAP cubes

andyblg — Wed, 30 Sep 2009 22:22:41 +0000

IBM review concepts “OLAP models” and “OLAP cubes” (to ibm’s page)

…”OLAP models and OLAP cubes are two critical concepts in DB2 DWE OLAP services. A cube model is built to represent a data structure and relationship in an OLAP data mart. A cube model contains metadata objects that describe relationships within the data that resides in the base tables and also describes where pertinent data is located. A cube model provides a new perspective for the information consumers from which to understand their data”…

…”An OLAP cube has a specific set of similar but more restrictive metadata objects derived from the parent cube model, including cube dimensions, cube hierarchies, cube levels, and a cube facts object. A cube can have only one cube hierarchy defined for each cube dimension, but a dimension can have many hierarchies that are defined for the cube model“…

TAG Cubes - SQLite Star Query Part III

Tom Gleeson — Tue, 29 Sep 2009 13:36:16 +0000

It’s no secret that I’m a huge fan of SQLite and Excel, particularly when used in combination. I also greatly admire the open source BI engines, Palo and Mondrian. Mondrian appeals because of its “ROLAP with a cache” architecture and its implementation of MS’s excellent MDX language. When I say MDX is excellent I’m talking with my professional programmer’s hat on, as an end-user tool it’s a non-runner. This is where Palo comes in, building on the hypercube concepts pioneered by the likes of TM1 and ESSbase, it presents a designview that’s approachable by a vastly greater percentage of “civilians” than is the case with ROLAP-based solutions.

The trick behind TM1, Essbase, Palo etc. is the extension of the spreadsheet metaphor from two to multiple dimensions, while still binding the interface closely to the familiar spreadsheet (which for most of the business world is still Excel).

So where does SQLite come in all this?

At first glance, SQLite lacks the sophisticated join functionality to support star-queries, but of course, if the dataset is small then a full-table scan of a fact table, or better still, loading the fact table into memory negates any such short-comings.

In fact, all traditional ROLAP engines have problems with dimensional models, particularly when you reach the point of using summary tables or query re-writes, that’s why the emerging SQL-speaking columnar-databases are such a godsend for ROLAP data warehouses.

It was SQLite combined with Excel acting as a data prep platform that was originally my main interest, so for pivoting, Excel’s own pivot table would have to do. Nevertheless, I felt the tool was incomplete without the ability to directly pivot the underlying SQLite database.

Why not use Palo or Mondrian as a pivot tool? Well yes, where a fixed permanent “solution” is required then the extra moving parts of either approach would be justified and indeed necessary but that is to miss the essence of what I call datasmithing.

Datasmithing is not data warehousing nor is it the provision of solutions (which, for example, Palo superbly enables in multi-user budgeting situations). Datasmithing, as a skill, is of course part of the process of both, but it’s on the edges, at perhaps the planning or consumption stages.

Datasmiths deal in the unknown, in change, in disaster recovery, in systems’ commissioning, in the never-ending barely-repeatable processes thrown up by daily business life. For that, the toolset required must be as simple as possible (but no simpler), self-contained, document-oriented, secureable (is that a word?) and easily archived and retrieved. Excel and file-based DBMSs such as MSAccess or SQLite fit the bill nicely, server-based technologies such as DBA controlled database servers or IT installed “solutions”, less so.

Jedox has made Palo relatively easy to install (and likewise, Canada’s SQLPower has made Mondrian setup a painless exercise via their excellent Wabit reporting tool), but, the zero-install, email friendly document approach that spreadsheets are famous (and infamous) for, is preferable in many situations. This is something that Microsoft have recognised in their Gemini add-in for Excel 2010, but Excel 2010 is a not here yet and it’s likely to be five years or more before it’s as common as Excel 2003 is today.

The inclusion of FTS full-text searching with SQLite triggered an ah-ah moment with regards to pivot-enabling SQLite.

The usual method that hypercube-like excel-friendly OLAP tools use to return data is via a UDF like so…

=DATA(“CubeName”,”value1″,value2″,…)

…where valueN represents dimensional elements, so…

=DATA(“SalesCube”,”Beer”,”Profit”,”Jan 09″,”Actual”)

…is the Actual Profit for Beer sales in Jan 09. The dimensional elements act as “tags” to locate a particular value, there is of course much more to tools like Palo; hierarchies, intra-cube rules etc. but in essence most OLAP tools are like www.delicious.com for number crunchers. This method of retrieving data fits well with how people use Excel and not just for pivots, but for embedding OLAP aggregated cells in lists. For example, a CRM scenario; a Sales Rep makes a list of her ‘best’ (subjective) customers, but needs hard (objective) stats, to be placed alongside the list to convince the boss or to track actuals against expectation.

Dimensional elements as tags; FTS3 virtual tables as fact table indexes; the concept of a TAG Cube was born.

In the above example “Profit” would most likely be described as a measure (Palo, a near pure hypercube does not distinguish between Measure and other Dimensional coordinates). Dimensions, measures and attributes are in reality interchangeable (a Customer ID can act as a dimension or an attribute, but by applying a Count Distinct to it, it’s a measure) but most OLAP solutions treat “Measure Dimensions” as different, and so do TAG Cubes.

By using the default fact table structure (a single-columned table) and querying using the default measure (which translates to the SUM() of that single value) a ‘pure’ approach can be used. But, ROLAP is tightly bound to the concept of a fact table, and since SQLite is relational, TAG Cubes offer the ability to use a wide fact table approach and I think gains considerably in flexibility by so going.

The above example of using Count Distinct, or the simple creation of calculated measures are examples of this flexibility. Another, is a measure based on SQLite’s concat_group aggregate function to provide a drill-down facility, e.g.

=DATA(“SalesCUBE”,”ROWIDList,”Beer”,”Jan 09″,”Actual”)

…where “ROWIDList” would be setup as concat_group(rowid,’,') and will return a comma separated list of the underlying fact table ROWIDs.

A major reason for rolling my own pivot engine was to add a concept of “namespaces” and to separate the implementation of these namespaces from the actual pivot. When a tag (or a predefined hierarchy of tags) is assigned to a cube, it’s also assigned to a namespace, in many cases namespace and cube would be synonymous, but in some cases a more sophisticated approach is required:

Multiple cubes sharing the same set of conformed dimensions would be best served by such cubes sharing a common namespace, and so they can.
Different consumers of the pivot may require the use of a different language, be that a spoken language or a different ‘business language’ e.g. Manufacturing Product Codes V Consumer Product Names. Again, easily done.
Sometimes identifying data can’t be shared with the datasmith or the numerical analyst working on a problem; in such cases being able to replace the actual namespace with an obfuscated one can be very useful. Or, for added security, the namespace might only be issued to approved PCs while the tag index and fact table are stored on a shared drive. Needs some more work to make managing such scenarios secure and easy to use but the structure is there.

As hinted on above, the three elements of a Tag Cube, the namespace, tag index and fact table can be assigned to different databases (i.e. files). Due to the wonders of SQLite’s ATTACH statement and the backup API’s ability to quickly load/unload databases in/out of memory, it’s possible, for example, to load namespace and tag index (i.e. the ‘dimensions’) into a memory database, while a very large (i.e. too big to fit to memory) fact table remains on disk. Fast and cheap SSDs will add further configuration options.

Although most of the TAG Cube functionality is available only within Excel, I’ve built a C based SQLite Virtual Table (cFact) to allow the tag index to used outside xLite. This means that SQLite drivers for ODBC (for use as a Pivot Table source, for example) or JDBC (for use in SQLPower Wabit perhaps) can efficiently access data models built using xLite.

I had to revert to using C rather than my preferred Python (did I mention that xLite now embeds Python in Excel, no, well it does, Python the newVBA ?), having failed to get around multi-threading issues with callbacks to Python in both the ODBC and JDBC drivers. I’d make a career promise to myself many years ago, not to having anything to do with printers or threads, and I think I’ll stick with it

TAG Cubes are the latest addition (still WIP actually) to be added to xLite, adding to:

VBA coded SQLite SQL functions.
Worksheet Functions; call out to a ‘function’ built using Excel formula, passing a parameter list and returning a value.
Workbook Functions; like Worksheet Functions, but loading a new Workbook, passing in parameters, passing back a value (or tables) and closing the Workbook when finished.
XLiteScript; xLite exposes its functionality via VBA coded UDFs, which can be called like any other formula, but data prep activities often require sequential procedural logic, xLiteScript is a table-oriented scripting mechanism offering basic flow-control logic.
pyScript; I embedded Python into xLite to take advantage of Python’s speed in developing Virtual Tables, SQL Functions and extensions to SQLite and to tap in the wonderful world of Python code. I’ve also added the ability to use Python from scripts defined within Excel (to indent, tab to the next cell!).
Fast load/unload to/from CSV.
Load from any ADO source.
Remove xLite formulae and rename and save Workbook, very handy when used via Workbook Functions to mass produce Excel “reports”.
Other WIP items are; load from SAP, load/unload to/from Amazon S3, use Palo cubes as TAG Cube “facts”, slot in/out Palo for TAG Cubes, auto-generate Mondrian XML based on TAG Cubes, write-back and splash, Python & VBA TAG Cube “rules”.

And what’s more, I intend to open-source the lot …

Why not join me on Twitter at gobansaor?

Database and cube dimensions

andyblg — Tue, 22 Sep 2009 07:42:12 +0000

Very interesting note from Microsoft (Sql Server BOL or Designing Dimensions):

“… A database dimension is a collection of related objects, called attributes, which can be used to provide information about fact data in one or more cubes. For example, typical attributes in a product dimension might be product name, product category, product line, product size, and product price. These objects are bound to one or more columns in one or more tables in a data source view. By default, these attributes are visible as attribute hierarchies and can be used to understand the fact data in a cube. Attributes can be organized into user-defined hierarchies that provide navigational paths to assist users when browsing the data in a cube.

Cubes contain all the dimensions on which users base their analyses of fact data. An instance of a database dimension in a cube is called a cube dimension and relates to one or more measure groups in the cube. A database dimension can be used multiple times in a cube. For example, a fact table can have multiple time-related facts, and a separate cube dimension can be defined to assist in analyzing each time-related fact. However, only one time-related database dimension needs to exist, which also means that only one time-related relational database table needs to exist to support multiple cube dimensions based on time…”

Thus:
- during designing DW we define database dimensions (concept of dw level)
- during designing Cube we have deal with cube dimensions (concept of cube level)

hm…

Offline OLAP - Working with Disconnected Analysis Service Cube

Benny Austin — Sun, 20 Sep 2009 13:55:30 +0000

Microsoft introduced a cool feature in Excel 2007 called Offline OLAP that enables you to wor

Data Warehouse vs Data Mart

sendtoshailesh — Mon, 14 Sep 2009 18:34:43 +0000

Data Warehouse

Data Mart

Scope & Application

Application Independent

A Data Warehouse is single point repository and its data can be used for any foreseeable application

Specific Application

Data-Mart is created out of a specific purpose. This means that you will have a data mart created to analyze customer value. This means that the designer of the data-mart is aware that the data will be used for OLAP, what kind of broad queries could be placed.

Domain Independent

The Data Warehouse can be used for any domain including Sales,Customer, operations, finance etc.

Specific Domain

A Data-mart is specific to a given domain. You will generally not find a data mart , which serves Sales as well as operations domain at the same time.

Centralized Independent

The control and management of data warehouse is centralized.

Decentralized by User Area

Typically a data-mart is owned by a specific function/sub-function.

Planned

Data Warehouse is a strategic initiative, which comes out of a blueprint. It is not an immediate response to an immediate problem. It has many foundation elements, which cannot be developed in an ad-hoc manner. For example, the standard sets of dimensions & measures.

Organic, possibly not planned

Data-Mart is a response to a critical business need. It is developed to provide gratification to the users, and given that it is owned & managed at a functional level, it grows with time.

Data

Historical, Detailed & Summarized

A good data warehouse will capture the history of transactions by default; even of there is no immediate need. This is because a data-warehouse always tries to be future proof.

Some history, detailed and summarized

It’s same with Data Warehouse. However, the level of history that is captured is governed by the business need. For example, a data warehouse will capture the changes in the customer marital status by default. A Data Mart may not do it, if Data Mart is created to profile/segment a customer on the basis of his spending patterns only.

Sources

Many Internal & external Sources

This is an obvious outcome of the Data Warehouse being a generic resource. That is also the reason why the staging design for a data warehouse takes much more time compared to that of a data mart.

Few Internal & External Sources

Self Explanatory- A limited purpose leads to limited sources.

Life Cycle

Stand-Along Strategic Initiative:

A Data Warehouse is an outcome of a company’s strategy to make data an enterprise resource. If there is any other trigger, chances are that it may not achieve its objectives

Typically part of a Business Project:

A Data Mart comes into being due to a business need. For example Risk Portfolio Analysis data mart could be a part of Enhancing Risk Management Initiative.

Long life

A Data Warehouse is a long-term foundation of an enterprise.

Can have any life span

A Data Mart starts with a given objective, and it can have a life span ranging from one year to endless. This is because some applications are core and business as usual to an enterprise. The life a data mart could be shortened, if a Data Warehouse comes into being.

Olap-Komponenten in Flex 3.0

manueldri — Sun, 13 Sep 2009 13:04:24 +0000

Vor ein paar Tage habe ich mich etwas über Flex schlau gemacht. Mit großen Erstaunen habe ich von den Olap-Komponenten gelesen, welche in Flex 3.0 enthalten sind. Denn Flex hat einiges zur Visualisierung von Daten eines Olap-Servers zu bieten. Unter anderem wird eine klassische Pivot-Tabelle sowie ein Chart angeboten. Im Netz befinden sich beeindruckende Beispiele zu den einzelnen Komponenten.

Sreenivas hat auf seinem Blog eine schöne Chart-Demo. Per Click kann an Drill-Replace ausgeführt werden. Per Ctrl + Click wird auf die übergeordnete Ebene angezeigt.

Ein Beispiel, dass sehr schön die Möglichkeit von Pivot-Tabellen zeigt, befindet sich auf Satishs Blog.

Die Kommunikation zwischen der Flex-Anwendung und dem Olap-Server kann momentan über XMLA erfolgen. Hierbei handelt es sich leider um einen veralterten Standard. So muss man gespannt warten, bis die Flex-Komponenten olap4j, den aktuellsten Standard zum Datenaustausch zwischen Olap-Clients und Olap-Servern unterstützt wird.

Viel Spass beim Stöbern.

Microsoft SSAS tutorial

sksastry — Fri, 28 Aug 2009 02:22:21 +0000

I am continuing to make an effort to learn something new everyday. I have worked a little on improving my T-SQL and OLAP development. I found this fantastic website, which has tutorials on microsoft SSAS. I read through a few of them these are really good. I am will post a link to the website. Anyone looking for a good introduction to Data warehouse, OLAP and other BI terminology and Microsoft SSAS, I highly recommend this website.

Until next time, continue to help others and learn something new every day

Agg Tables a different look

mgcuthbert — Thu, 27 Aug 2009 18:33:46 +0000

Aggregate tables are a great performance booster for any reporting system. It has the ability to really shave seconds of the length of time it takes to return results. However never rely solely on agg tables as it is not substitute for good database optimization, especially in regards to database indexes and statistics, it also does not replace any reporting caching layer that you may build into your system. It is simply another piece of the puzzle. This post is more of an different kind of look onto agg tables, not how to build them but rather what to build and why. For a detailed more technical explanation of agg tables, the Mondrian website I think has a good article for that here.

So the strong question is firstly why use aggregate tables at all, and to give you a basic answer to that question let us for a second say that you are tracking page views on your website. Each person comes in gets a track call that enters a record into your database. My report simply is a basic line graph that shows the total number of page views per day. So if we have a million page views over 1 month our calculation has to go through 1 million entries in our fact table and add up all the values per day. We would be doing a lot of extra work adding up all these values every time we requested that report. Now let us say instead we decided to aggregate that data per day. So instead of say roughly 30,000 records over a single day we aggregate that data into a single record. So we have another table that tells us that for Day 1 we have 30,000 page views. In doing this we now are doing our calculation for page views per day on a table that will never have more records then the number of days you are requesting. This is always going to be a lot faster than if you did not have any aggregate tables at all. Granted that the original type of query shouldn’t have taken too long anyway but this is to give you an idea of why agg tables are good.

Of course reporting is rarely that straight forward and usually their are multiple variables that you would be reporting on. Each variable possibly increases the size of a aggregate table and also the number of aggregate tables that you could create. For example from our previous example we were just tracking page views per day. So that is basically 2 variables making the maximum records only ever N, where N is the number of days. However what if you wanted to report on page views per url per day. Now all of a sudden your maximum records in the agg table is NxM where N is the number of days and M is the number of urls that were hit. At some point the number of records in your aggregate table could get a little too close to the actual number of records in your fact table, and hence defeating the whole point of actually having aggregate tables to begin with. It is important to remember that aggregate tables should be used more like snapshots of data instead of detailed data, which need to be optimized generally in other ways.

The other problem is the number of possible combinations of agg tables that could be created for your reporting engine. It can understandably become very troublesome and unmanageable. Firstly it is important to understand that aggregate tables are not there to solve every reporting variable (although I guess technically it could). It is best to have a good understanding of how your reports are being used, which reports are used the most, what sets of data are most requested and which are not. Once you have a good basic understanding of how exactly your reports are being used (or going to be used) you can then start deciding what type of aggregate tables you can create to help. I have found that the easiest way to manage this is to have your reporting engine build the aggregate tables for you, however this often requires a little more technical know how and would required dynamic schema’s and managing rollups of data efficiently and effectively in the background while your system is running. Maybe I will write a little bit more about how exactly I have accomplished that task in the past in some other post.

Hopefully you are a believer in aggregate tables, and hopefully you also understand that it is important to understand the requirements of your system before wasting a lot of time and effort on building tons of different types of aggregate tables only to find that it is used maybe 1% of the time. When building reporting systems in general I feel it is vitally important to build a solid framework and understand exactly what the reporting users want to get out of the reports so that you can effectively solve their problems, and when using Mondrian aggregate tables are simply one important part of that puzzle.

Entering the Blogging world

sksastry — Thu, 27 Aug 2009 02:50:14 +0000

I have finally decided that I need to start blogging. It may sound lame, but I have started to blog my ups and downs in my job search. I recently found out that the project that I am working on is being terminated(Thanks to the economy) now I am back in the job market.

I have started actively looking for a job since 2 weeks and have not much success although I did get to meet a recruiter.

At this point I would like to write a little about my skill set, I have been working on databases since the beginning of my career. I have worked on Lotus Notes (yes it is a database system for those of you who think other wise) For the last couple of years I have been on working on SQL and Postgres. And in my latest project I got into developing OLAP models. This has been the most exciting part of my recent position. I like the fact that the we can develop data models to view the same data in variuos different ways. It is really cool, I worked mostly on Open source software for data modelling and report generation, I know that Microsoft and Oracle are the big players in the field and I am still learning.

I’ll also use this blog to help in my learning process. And may be at some point publish some data models

That’s it for now

OLAP a start.

mgcuthbert — Fri, 21 Aug 2009 20:29:36 +0000

So over the last couple of years I have spent a lot of time working with Mondrian. Mondrian is basically an OLAP server written in Java, and one of many options out there with the biggest probably being Microsoft’s Analytics Server. However there are definite financial reasons for choosing Mondrian over Microsoft. Mondrian was bought by Pentaho a while back and brought in to be a part of their BI suite. But it has stayed fairly independent of the suite as a whole, which for me at least seems a good thing as I didn’t enjoy the BI suite as much. I am sure some love it, but I had to debug code a lot in Mondrian and some of the BI packages which I did not really enjoy. Mondrian just seemed to have a much cleaner code base, or at least that was my impression of it. I haven’t looked at it in a while though so I am sure things have changed. So firstly why even the need for an OLAP server, would a SQL database do the trick? And the short answer is yes and no. A SQL server (postgres, MySQL, MSSQL, Oracle, etc. etc.) definitely does the trick and ultimately an OLAP server simply is an extension of a SQL server, it gives it that extra added dimension (excuse the pun). OLAP basically allows you to view your data in 3 dimensions as apposed the basic 2 dimensions that you get in a standard relational database. This generally is of most use when building a reporting engine and the data you are reporting on has multiple aspects to it and multiple ways of viewing that data. Obviously if your data is very simplistic and 2 dimensions are more than enough, then no need to worry.

So I thought it would be useful to write about my experiences with building a reporting engine based on the Mondrian OLAP server. Most of my discussion will focus exclusively on Mondrian itself and not the reporting engine. The reason I thought I might do this is because during my learning process I found very little information about OLAP servers and particularly Mondrian. There was a fair amount of information about MDX (which is like SQL for OLAP) but I felt I still had to experiment a huge amount just to get things working. So to give you an idea of topics I will cover here is a list of the top off my idea that I will almost certainly talk about.

Aggregate Tables – Super super important in any OLAP schema, and I will talk about distinct values in this schema as that caused me the most headache in trying to explain certain things to people.
Performance Issues – Directly related to Agg tables but deserves a topic all on it’s own. How do you manage databases that will contain millions of records efficiently and quickly.
MDX / OLAP Schema – Little tips and tricks I found along the way and various ideas behind building effective schemas.
XMLA – How to get your data to the outside world.
Integration – There are many issues that one has to consider when integrating a OLAP solution into their reporting engine.

And really there are so many topics that I can talk about with this, but that is the general idea. I will try to have one blog entry per week on Mondrian, however who really knows how much I will end up talking about it. I also don’t think I know everything, far from it actually there is still so much I don’t know and there may be things that I am sure I am doing incorrectly, but this has been my experience. The first topic however that I will write on is Aggregate Tables, as these are an especially fun topic for me. Unfortunately for inexperienced developers this may not be that helpful initially as I will assume a fairly decent knowledge of schema development. I will post that up hopefully early next week sometime.

What do you think about definition of OLAP?

andyblg — Wed, 19 Aug 2009 19:40:38 +0000

What is OLAP?

I sought formal definition of OLAP. It’s wery interesting.

1. Nigel Pendse “What is OLAP?” (http://www.olapreport.com/fasmi.htm)

2. http://olap.com (http://olap.com/w/index.php/OLAP_Education_Wiki)

3. OLAP and related terms (http://keywen.com/en/OLAP)

4. OLAP definition (http://www.yourdictionary.com/computer/olap)

5. OLAP glossary (http://pagesperso-orange.fr/bernard.lupin/english/glossary.htm)

I think it is quite enough.

Finally “What is OLAP?”

- OLAP is a tool or technology of data processing (?)

- OLAP is a tool or technology of data visualization (?)

- OLAP is a tool or technology of data representation (?)

- OLAP is a class of system which include the above (?)

- OLAP is a class of system which include the above or something from this or something else (?)

and… What do you think about formal definition of OLAP?