Thursday, December 23, 2010

Pivot table: OBIEE 11g Vs Qlikview

With the launch of new version of Oracle BI OBIEE 11g, some new features and enhancements got added.
I installed the new version and started looking at the new features. This is completely a new experience. The look and feel is very much improved now and is much better.
Here in this post I will discuss only about the new feature introduced in Pivot charts. Now the pivot has the ability to expand and collapse the levels using the + and - signs as shown below.
For those who have used or worked only on OBIEE before it will be a new way of interacting with a pivot but even for some niche tools like Qlikview, this is a very basic feature.
If we just compare this new feature of OBIEE11g with Qlikview from end user perspective, it stands nowhere.
Now lets discuss how we can achieve this in OBIEE11g and then we will compare it with Qlikview:
1. You first need to create a hierarchy in the presentation layer: For example, I have created a presentation hierarchy named "Products" having 3 levels "LOB" - "Type" - "Prod Name" as shown below.
Now when a report developer or user go to the Answers to create a pivot, we need to select this Hierarchy in the pivot as shown below.

Now if you run the report you will see the results like this:
Till here it was good but just think a user would just want to have just 2 levels "LOB" and "Prod Name" in the present pivot. To achieve that, the user needs to ask the developer to create a new presentation hierarchy with only required two levels or make modification to the existing presentation hierarchy and remove the "Type" level.
On the similar lines if a user require this pivot to have a column from some other dimension say for example "Time". If we just add a column from the "Time" dimension lets say "Year", the pivot will look like this:
This is definitely not what we need.
So we have seen the limitations and now we will see how easily an end user can create such reports without any help from developers or IT.
Simply create a Pivot table report and select the dimension columns and fact column. Here in this example I have selected the 3 columns which has been used to create the pivot in OBIEE11g. This is how the report will look like:

This is really simple and no hierarchies needs to be created anywhere. Now we will try to add some Time dimension column in the existing pivot and see how it looks. Simply add the column in the dimensions tab and arrange by promoting or demoting to see it wherever we want. I have added it to the top and this is how it will appear:
This is really very simple and effective and the users really find it easy to see how they want to see it.
Going one step ahead lets say I want to have the subtotals of levels to be displayed in the pivot, Qlikview does it by simply selecting some of the available options. In Properties-Presentation tab there is a check box which says show Partial sum. You can also choose for the level for which you require the partials sums above or below. In OBIEE11g, I could not find anything other than the grand total sum. This is how it finally looks in Qlikview:
This is really useful. Also there is no option of Expand All for the entire pivot or for a selected level. You can just expand a particular level value that means you needs to click on all the + signs one by one to expand the pivot whereas in Qlikview you can expand the complete pivot or a particular column or level.

These small small options may not sound that important to some of us but for an end user it makes the entire experience interactive and helps him feel more independent and brings the feeling of ownership.
Till next time...

Manohar Rana

Monday, December 13, 2010

Column Selector feature in OBIEE and Qlikview

In one of the projects, there was a requirement to have column selector. There should be some fixed columns in a report and some columns which the user should be able to change by simply selecting the column from a list and not going to the design mode or ad hoc mode.
Here we will see how we did it in OBIEE using the column selector feature and how we can do it in Qlikview.

Lets first create a simple request in OBIEE Answers as shown below. I have selected two dimension column and a fact column here. Click on Add view and from the list select Column Selector.

Now you will see the columns you have selected in column selector mode as shown below.

Select the check box below column 1 and click on the columns from the list of columns available in the left. Similarly do it for column 2 by clicking on the check box and selecting columns. Uncheck the check box which says" Automatically refresh when a new column is selected". Click on OK and you will see the column selector present in your compound layout.

Now you can check by selecting different columns and clicking on Go button to see that the columns in the reports gets changed.
You can also have some columns fixed and some columns changeable. To do so, create a dummy column "None" and make it hidden in the column properties as shown below.

Now go to the Column selector view and select the columns you would like to see in the list as shown below.

Click Ok and come to the compound layout and you will see only one column which is a fixed one. In the column selector, you will notice that None column is selected. When you select any other column from the column selector, the new column will get added to the report as shown below. If you again select 'None' column, it will again hide.

Isn't it a nice feature and very easy to use for end users and equally easy to develop.

Now we will try to implement the same feature in Qlikview and see to what level we succeed.
Create a simple report in Qliview again using 2 dim columns and one fact column.

Now lets create two Groups (Cyclic) and add the columns required in these groups as shown below.

Now, select these two newly created cyclic groups in the report as shown below.

Click Ok to see how it works. Just close to column headers, there will appear cyclic icons. When you click on this icon, the list of columns will appear and when selected the column will replace the existing column.

This is also very simple. But I did not find anything to make some columns fixed and some columns changable. I tried creating a dummy column and added that dummy column in cyclic group. But as the these groups are embedded in the report itself and not a seperate entity as in OBIEE, the dummy column when selected shown all blank rows as shown below.

This would definitely not be acceptable to the users. So I tried to make a column hidden based on a condition but that made the cyclic icon hidden alongwith the column hence the user can not select any other column from that group.
You can hide any column based on certain condition as shown below but it did not served the purpose here.

I am not sure if there is some way to accomplish this but with out of the box features and not by writing some macros.
If anyone has any pointers, please do suggest.

Manohar Rana

Saturday, March 13, 2010

BI Project Implementation Life Cycle


Recently I went through the website of Qliktech and read where it says "And unlike traditional BI, QlikView delivers immediate value with payback measured in days or weeks rather than months, years, or not at all.". It made me think for a while. what are they referring to ? Do they mean Qlikview as a BI solution can be implemented in weeks? Are they talking only about small department level deployments or just the development phase of the BI project life cycle or is it just a sales pitch? I am just wondering how much part a tool plays in a BI project Implementation life cycle and how does a tool effect the development phase.
Whether you use Qlikview or some traditional BI like OBIEE for that matter, about 70% of the project life cycle remains unaffected or untouched.
Then I thought about my recent Oracle BI Project. If we would have replaced OBIEE with Qlikview, will the whole project gets implemented in weeks instead of months or years. I don't think so.

Well, The topic is BI Project Implementation Life Cycle and not Qlikview.
I have worked on some small Qlikview project Implementations which generally span from 2-4 Months and also large OBIEE implementations ranging from 9 months to close to 1 year.
My latest assignment was a large Oracle BI (OBIEE) Global Implementation Project.
I would like to share my knowledge and learnings from the current as well as previous projects. I will also discuss how large deployments differ from smaller ones and also the different phases involved in a BI project.
Big deployments generally includes ETL as well as BI and takes somewhere around 9 months to 2 Years to implement the solution. But when we compare it with small deployments using tools like Qlikview, it has a small life cycle ranging from somewhere around 3 months to 6 months because they generally do not involve a datawarehouse.
There are various reasons for not having a datawarehouse like

1. Time and Cost: For smaller organisations the IT budgets are generally small. Including Datawarehouse in a BI initiative increases the project cost significantly. The organisations needs to Purchase additional license for a new database. Even if they already have a database in place, it can not be used for various reasons and is not recommended as well. The organisation also needs to purchase a license for a ETL tool and required hardware. Other than the software and Hardware, the cost is involved in hiring resources for database and ETL.

2. Small data size: As small organisations do not have large or huge data sets (Even if they have multiple applications) and hence generally neglect the relevance and importance of having a datawarehouse in place. Not only it complements BI by providing faster response times for BI end users but also it reduces the development cycle of the BI by reducing the level of complexities of the logic. Using a datawarehouse helps in having much simpler BI data models which requires less development time and maintainance.

3. Less complex data models: Small organisations generally have small customised business applications running which do not have very complex OLTP data structures and hence it becomes relatively easy to design BI data models eliminating the need for a datawarehouse solution.

4. Less number of Users: Organisations having smaller number of BI users (Using the application at a given point of time) generally have this option of building their BI solution directly on top of their OLTP database. They can either allow BI users to directly query the OLTP database which in some way reduces the performance of the business application users already connected or they can use tools like Qlikview or Hyperion Essbase which employs this technique of storing the data in their proprietery data file formats and allow a disconnected type analysis. The later one is a much better option as it provides a much faster performance as against querying an OLTP database as for some reason unknown to me these proprietary data files are highly optimized to be used by their respective applications and also this do not impact the performance of the OLTP application.

Now let's discuss a typical BI Project life cycle which comprises of following phases:

1. Project Initiation and Analysis:
For any project to start a business need is a must. for example "replacing an old customized reporting system with a new reporting and analysis system which is more flexible, cost effective, fast and requires less maintainance" or "Need to consolidate data from disparate sources and a common standardized platform for reporting and analysis" could be a business need.This business need is evaluated at a very high level as to how critical is the need and how it well it falls in line with the strategic objectives of an Organization. Formally a Project manager is identified and appointed who further investigate and perform some first level of analysis.
It is then Project manager who creates a document with details like Business case, initial risk assessment, scheduling and budgeting, stakeholders identification etc taking help from the available resources and then a formal approval is taken.

2. Planning and Designing:
Requirements gathering is the most important and critical part of the whole project. A requirement document is created for this purpose and all the requirements are documented in much details. The requirements are finalized and then project scope is created. To fulfill these requirements, various tools are evaluated and whichever is the best fit is selected. Now the team resources are identified. All the logistics, hardware and software requirements are identified and procured.
Data models are designed and it is documented in a Technical design document which also specifies other technical details like features and functionalities of the design, security and authentication, best practices, repository variables, layers, database connections, connection pools etc etc..
Prototypes are designed to show the features and functionalities as documented in the requirement document and is formally approved. Project kickoff.

3. Development:
Before starting the development of the data models and reports, the project scope is divided into small more manageable modules or batches.
It is a good practice to divide it on the basis of functional areas and subject areas.
So let's suppose we decided to do it for functional area Human Resources among other functional areas like Sales and Distribution, finance, Inventory and Manufacturing etc under which we are planning to create different subject areas like

Employee Compensation and Bonus, Learning and Development, Employee movements and transfers etc. You may have one data model for complete HR or seperate data models for each subject areas.
For smaller organizations with less complex OLTP data structures, it is possible and feasible to have a single data model for complete Human Resources.
For large and complex OLTP structures, it is generally not possible as otherwise the size of the fact table will be extremely large horizontally as well as vertically. This will give an extremely slow performance as well as from maintainance perspective also the time taken to load the fact table will be more and unpractical.
Once we decide on our strategy for the development, we start with developing the data model as per the designs created in the previous phase i.e Planning and Designing.
The data model is developed generally in layers. In Oracle OBIEE, there are three layers and Cognos allows you to build as many layers as you wish and BO provides 2 layers(I am not very sure on this and would request some comments on this).
In Qlikview, we can make it single layered or 2 layered by renaming the column names in the script.
For all practical purposes, upto 3 layers is a good idea but you may agree or disagree on that. Based on your
requirements of maintainance you can decide on that.
OBIEE has 3 predefined layers namely Physical Layer, Business and Modeling layer and Presentation layer.
Physical layer is where we simply make connections to the database and import the metadata for database objects like tables, columns, views, primary and foreign key relationships. Now we do not make any changes related to changing the names of the columns which help the administrator and developers from maintainance perspective.
Based on the available physical layer objects we create our OLAP models in Business layer by defining dimensions, facts and Hierarchies.
In the presentation layer, we categorize the objects based on subject areas from the objects available in OLAP model in Business layer. We rename the objects present in Presentation layer from end users perspective or business terminology.
This whole process really helps the developers to understand and visualize the complete model and saves lot of time in debugging or day to day maintainance activities. This process oriented approach is again an attempt to divide and rule and making our life a bit simpler.
Once you are done with your model, the next step is to start developing your reports and bringing in the identified resources for report development into the team.
The reports based on the subject area are divided among the team and with the help of report specifications available in Technical design document created in the previous phase.The reports are generally designed by report developers.
While the report designers are engaged, the data model developers work developing the next subject area. Initially the team size is less and as the work keeps on growing, more people are added in the team.
The development also includes setting up the object level as well as data level security , creating users and groups and creating variables as per the technical design document.

4. Testing
Testing is one of the most critical phase and also sometimes most time consuming phase of a BI project Implementation.
There are 3 types of testing done on a project namely Unit Testing (UT), System Integration Testing (SIT) and User Acceptance Testing (UAT).
Unit Testing is generally done by the developers and they test that the code or report they have developed is working as per the requirement or specifications. This is performed in Development environment where Developers had developed the
report. Developers prepare a test case plan for themselves listing all the cases they would like to test. These cases could be testing the font size, color, spell check, prompts, filters, data values etc.
Sometimes developers may exchange the reports with their team members to perform a peer unit testing and this is a good practice as it is little easier to find out mistakes in other's report than your own.
Once Unit testing is complete, the code (data models and reports) are transferred to Test Environment.
This Testing Environment is generally similar to the production environment but that is not the case always. Having the test environment same as production environment allows us to anticipate the performance and behavior of the developed solution in a much better way.
Once the code is transferred to Test environment, System Integration Testing is performed. SIT checks how all the individually pieces works collectively and are integrated well to produce the desired output. This test is performed by the IT team members or by identified testers from the client side. However before they perform the test a sampling based dry run is required to be performed by the development team.
Once the testing team start testing the application, they put all the defects in a defect log sheet mentioning the
details of the defect.
At this point of time, it is recommended to appoint some dedicated members from the development team to fix those identified defects and update the defect log sheet. While this activity is going on, other team members are assigned next set of development work and they keep working on developing next batch of reports. It may happen that the same team will fix the defects by allocating some portion of their time and rest of the time in developing next batch reports. But this may bring some imbalance or turbulence in the system as it will become very difficult to really work on two things simultaneously. Bug fixing involves lot of coordination with ETL team as well as testers and sometimes consumes time more than what was anticipated which ultimately may impact the development activities. Having a dedicated team for bux fixing activities would be very useful and effective.
Once all bugs are identified and fixed, the Business users are asked to perform the User acceptance test. The test cases are prepared by business users and they check if all the requirements are fulfilled and they really getting what they want. Here business users compare the result set with the result set from their existing system and verify the results.
One of the biggest challenge in SIT or UAT is if any data related inaccuracies are found, it becomes really difficult to find the root cause. The developer needs to check which version is true. There may be some internal logic or transformations or formulas applied in the existing application and this analysis consumes whole lot of time requiring lot of coordination with the team supporting the existing application or system.

5. Implementation
After the code is tested and verified completely, it is transferred to the production environment and opened for the real end users. Before that general awareness sessions and training sessions are held for the end users to use the new system. For some time the new system is put on a stabilization period (Generally ranges from 15 days to 2 or 3 months but it could be even more) where in case a bug occurs, it is fixed by the same development team.
During this time the new system or application is made to run parallel with the existing system or application and once the stabilization period is over, the old system is replaced partially or completely.
Once the stabilization period is over and the system gets stabilized, the support team is provided with all the project related knowledge and the development team is rolled off.
The implementation of BI Project gets over.

Please feel free to share your comments.

Manohar Rana