I’ve been thinking through and experimenting a bit more with some of the concepts in SQL Azure – specifically I was considering the impact of fragmentation on both the storage (in terms of the storage limit) as well as the maintenance. This is not a new issue, DBA’s face fragmentation regularly and can deal with it in a variety of ways, but with SQL Azure the problem looks magnified by a lack of tools and working space. Whilst looking into this, I then realised that there is an unfortunate consequence of not knowing how much data space your index is actually using.

Each table in SQL Azure has to have a clustered index if data is going to be inserted into it and clustered indexes can suffer from fragmentation if chosen poorly. The combination of SQL Azure and the time-honoured fragmentation provides three consequences about in, fragmentation:

• will occur and you have no way in which to measure it due to the lack of DMV support.
• will create wasted space within your space allocation limit.
• will reduce your performance.

You could work it out if you knew how much space you had actually used vs. what the size of the data held is, but we are unable to measure either of those values. If you have chosen the data compression option on the index then even those values would not give you a fragmentation ratio.

This leaves us with a situation in which you can not know how much you are fragmented, meaning:

• You schedule a regular index rebuild.
• Hope SQL Azure performs index rebuilds for you.

I’m not aware of SQL Azure doing this for you – and you do not have SQL Agent facilities either.

So this seems very wrong, the concept of SQL Azure is to take away a lot of the implementation details and hassle from the subscriber – DR and failover is handled etc. But there looks to be as gap in which certain items such as fragmentation is falling – I have not seen any documentation saying SQL Azure handles it (but there could be some hidden somewhere and I hope there is!) and neither are you given the right tools in which to program and handle it yourself.

What happens when you hit that size limit?

Msg 40544, Level 20, State 5, Line 1 The database has reached its size quota. Partition or delete data, drop indexes, or consult the documentation for possible resolutions. Code: 524289 

That took a lot of time to get to, (SQL Azure is not fast), but was generated using a simple example that would also demonstrate fragmentation.

Create Table fragtest ( id uniqueidentifier primary key clustered,
padding char(3000)
) 

Very simple stuff, deliberately using a clustered key on a GUID to cause a decent level of fragmentation, and also using the padding fixed with character field to ensure 2 rows per page only, maximising the page splits.

insert into fragtest values (newid(), replicate('a',1000))
go 200000

Because of the randomness of the newid() function, the level of fragmentation is not predictable but will certainly occur – in my test I hit the wall on 196,403 records and failed with an out of space message.

Given the 2 rows per page and the number of rows, with ~0% fragmentation the data should be able ~767Mb – that is considerably short of 1 Gb – so there is a significant level of fragmentation in there wasting space, about 23% of it. If you include the 2k per page being wasted by the awkward row size then the actual raw data stored is roughly ~60% of the overall size allowing for row overheads etc.

So there are two important points from this contrived example:

• You can lose significant space from bad design.
• Doing this backs you into a corner that you will not be able to get out of – this is the worst part.

How are you cornered? well, try work out how to get out of the situation and defrag the clustered index / free up the space, you could:

• Attempt an index rebuild.
• Try to rebuild it with SORT_IN_TEMP.
• Drop the index.
• Delete data.

The first three fail, the SORT_IN_TEMP is not supported and would not of rescued the situation either since you have no working space in which to write the newly sorted rows prior to removing the old ones.  So do you really want to delete data? I don’t think we can consider that an option for now.

This all seems like a ‘rock’ and a ‘hard place’; whilst SQL Azure can support these data quantities,  it seems prudent that you never consider actually going close to them at all – and that you equally are going to find it difficult to understand if you are close to them, since there is no way of measuring the fragmentation. The alternative is that you manually rebuild indexes on a regular basis to control fragmentation, but then enough free space is going to have to be left to allow you to rebuild your largest index without running out of space – reducing your data capacity significantly.

The corner is not entirely closed off, the way out of the corner would be to create another SQL Azure database within my account and select the data from database1.fragtest to database2.fragtest and then drop the original table and transfer it back – not ideal but it would work in an emergency.

I think the key is to design to make sure you do not have to face this issue; keep your data quantities very much under the SQL Azure size limits, and watch for the potential of tables being larger than the remaining space and preventing an re-indexing from occurring.

Interested to know your thoughts on this one, and what other consequences of being close to the limit will come out.

You’re on your way to PowerPivot for SharePoint functionality – you’ve uploaded your PowerPivot for Excel workbook to your SharePoint PowerPivot Gallery.  You view the thumbnails of your report and they look nice.

From the thumbnail, you click on the report you want to see, and the report renders nicely.

But then you click on a slicer, and then all of a sudden you get an error like the one below.

 What can you do? 

Read more…

In this posting we will take a more detailed technical look at how the data refresh facility works and the steps that it takes to accomplish a data refresh cycle. Rather than starting with the “Manage data refresh” page, we will assume that you know how to setup a schedule – in this posting, we will take a deep dive into the cycle itself.

What steps are taken when the data is refreshed?

Now that you have configured your schedule(s) for the workbook, let’s take a step back and examine more closely what data refresh actually means. I think that it is valuable to understand, at some basic level, exactly what the system is going to do on your behalf at 2am in the morning. When a job actually run, the data refresh facility goes through the following steps:

Read more…

Well, I have a fun one today. We get asked all of the time about how the PowerPivot add-in works. Particularly how it relates to the SSAS components that BI Pros are used to, e.g. making connections to SSAS. As I am getting ready to write an architecture overview blog posting for the PowerPivot team blog, I decided to take the client architecture out for a spin and attempt a more detailed drilldown here . . .

Here is a block diagram of the PowerPivot client component architecture

Read More…

I got something wrong on a couple of postings recently, namely the relationship between BLEVEL and the number of blocks needed to read “down” an index, the true depth or HEIGHT of the index {I used to know this but I forgot, but heck no one pinged me on the two posts in question, so I got away with it – I’ve updated the postings already.}

BLEVEL is the number of branch levels (including the root node) in a B-Tree index. Height is the actual depth of the index. Height is BLEVEL plus one. So when you see BLEVEL of 3 against an index in DBA_INDEXES/DBA_IND_STATISTICS, that means the index has a root node, a first level of Branch blocks, then a second level of branch blocks and finally the Leaf blocks (that hold the indexed values and rowids to the table entries).

Thus to scan the index for one unique entry, Oracle will need to read the root node to locate the correct branch node in branch level one, read that to find the correct branch node in branch level 2 and that will lead to the correct leaf block. That is four blocks to read. The leaf block contains the index entry and the rowid of the relevant data block, which allows oracle to go directly to that block, for the fifth block read.

{I’m having trouble finding a nice diagram of this {{ I hate the one in the Oracle manuals}}, not even on Mr Foote’s or Mr Lewis’s pages, so if you spot one before I do, let me know and I’ll update this page with a relevant link}.

Some documentation on the Web mentions HEIGHT being held in the index stats table. This is SYS.INDEX_STATS, not the DBA_IND_STATISTICS table, and SYS.INDEX_STATS is only populated when you run the old “ANLAYZE INDEX index_name VALIDATE STRUCTURE” command, so ignore that.

The below demonstrates the increasing BLEVEL and the number of consistent gets to select one record {it’s more complicated if you select more than one}

NAME                           VALUE
------------------------------ --------------------
compatible                     10.2.0.3.0
cpu_count                      8
db_block_size                  8192
db_file_multiblock_read_count  16
optimizer_mode                 ALL_ROWS
sga_target                     0
sort_area_size                 65536

create table test_bl
 (id    number(8) not null
 ,status number(1) not null
 ,num_1     number(3) not null -- random 20
 ,num_2     number(3) -- random 20
 ,num_3     number(5) -- cycle smoothly
 ,num_4     number(5) -- cycle smoothly
 ,vc_1      varchar2(10)
 ,vc_2      varchar2(10)
 ,vc_pad varchar2(2000))
 tablespace users
 /
Table created.

insert into test_bl(id,status,num_1,num_2,num_3,num_4
                   ,vc_1,vc_2,vc_pad)
select rownum,decode(mod(rownum,100),0,1
               ,0)
,trunc(dbms_random.value(1,20))
,trunc(dbms_random.value(1,30))
,mod(rownum,10)+1
,mod(rownum,100)+1
,dbms_random.string('U',10)
,lpad(chr(mod(rownum,6)+65),5,chr(mod(rownum,6)+65) )
,lpad('A',100,'A')
from dba_objects
where rownum < 500
/
499 rows created.

commit;
Commit complete.

-- now add a pK on the ID
alter table test_bl
add constraint tb_pk primary key (id)
using index
tablespace users
 /
Table altered.

begin
  dbms_stats.gather_table_stats(user,'TEST_BL');
END;
/
PL/SQL procedure successfully completed.

select index_name,blevel,leaf_blocks
from dba_indexes
where owner=user
and index_name like 'TB%'
/

INDEX_NAME                         BLEVEL LEAF_BLOCKS
------------------------------ ---------- -----------
TB_PK                                   0           1

So I’ve created the test table, put 499 records in it and added the index, via a primary key constraint. The index created has one leaf block in it and a BLEVEL of 0.

Now let’s select a record via it {and the reason I put 499 records in the table is so that oracle decides to use the index and not a full table scan, which would be a danger with a very small table}.

set autotrace on
select vc_1 from test_bl where id=54
 /

VC_1
----------
BRFVRHEMWP
1 row selected.

Execution Plan
----------------------------------------------------------
| Id| Operation                   | Name    | Rows  |Bytes| Cost|
----------------------------------------------------------
| 0 | SELECT STATEMENT            |         |    1 |  15 |  1  (0)|
| 1 |  TABLE ACCESS BY INDEX ROWID| TEST_BL |    1 |  15 |  1  (0)|
|*2 |   INDEX UNIQUE SCAN         | TB_PK   |    1 |     |  0  (0)|
----------------------------------------------------------

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
          2  consistent gets
          0  physical reads
          0  redo size
        339  bytes sent via SQL*Net to client
        338  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed

select vc_1 from test_bl where id=54
 /

VC_1
----------
BRFVRHEMWP
1 row selected.

Execution Plan
----------------------------------------------------------
|Id | Operation                   | Name    | Rows |Bytes| Cost
----------------------------------------------------------
| 0 | SELECT STATEMENT            |         |    1 |  15 | 1   (0)|
| 1 |  TABLE ACCESS BY INDEX ROWID| TEST_BL |    1 |  15 | 1   (0)|
|*2 |   INDEX UNIQUE SCAN         | TB_PK   |    1 |     | 0   (0)|
-----------------------------------------------------------

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
          2  consistent gets
          0  physical reads

I generally run test selects twice, to remove any parsing and recursive SQL overhead {I’ll remove these from the rest of this post}. So, 2 consistent gets. That would be one on the index and one on the table then.
Note the cost of the index index unique scan – 0. See end.

Now I’ll add more data and grow the index.

test102>set autotrace off echo off
insert into test_bl(id,status,num_1,num_2,num_3,num_4
                  ,vc_1,vc_2,vc_pad)
select rownum+500,decode(mod(rownum,100),0,1
              ,0)
,trunc(dbms_random.value(1,20))
,trunc(dbms_random.value(1,30))
,mod(rownum,10)+1
,mod(rownum,100)+1
,dbms_random.string('U',5)
,lpad(chr(mod(rownum,6)+65),5,chr(mod(rownum,6)+65) )
,lpad('A',100,'A')
from dba_objects
where rownum < 5500
/
5499 rows created.

begin
  dbms_stats.gather_table_stats(user,'TEST_BL');
END;
/
PL/SQL procedure successfully completed.

select index_name,blevel,leaf_blocks
from dba_indexes
where owner=user
and index_name like 'TB%'
/

INDEX_NAME                         BLEVEL LEAF_BLOCKS
------------------------------ ---------- -----------
TB_PK                                   1          11

So now we have a BLEVEL of 1 and 11 leaf blocks. That will be a root node and below it the leaf blocks. Let’s try a select:

select vc_1 from test_bl where id=454
/
VC_1
----------
IQGSEOCCCH

Execution Plan
----------------------------------------------------------
|Id | Operation                   | Name    | Rows  |Bytes| Cost|
----------------------------------------------------------
| 0 | SELECT STATEMENT            |         |     1 |  11 | 2   (0)|
| 1 |  TABLE ACCESS BY INDEX ROWID| TEST_BL |     1 |  11 | 2   (0)|
|*2 |   INDEX UNIQUE SCAN         | TB_PK   |     1 |     | 1   (0)|
-----------------------------------------------------------

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
          3  consistent gets
          0  physical reads

3 consistent gets, one for the root node, one for the relevant leaf block and one for the data block holding the record.

Now I’ll insert about 400,000 more records to cause the index to become one level deeper. You might be interested to know that 300,000 records was not enough to cause a layer of branch nodes to be created, though as I am indexing an ascending numerical column, each index entry is not exactly huge.
I will then select my record:

99999 rows created.
99999 rows created.
99999 rows created.
99999 rows created.

begin
  dbms_stats.gather_table_stats(ownname=>user,tabname =>'TEST_BL'
                                ,estimate_percent=> 10);
END;
 /
PL/SQL procedure successfully completed.

INDEX_NAME                         BLEVEL LEAF_BLOCKS
------------------------------ ---------- -----------
TB_PK                                   2         761

set autotrace on
select vc_1 from test_bl where id=454
 /

VC_1
----------
IQGSEOCCCH
1 row selected.

Execution Plan
----------------------------------------------------------
|Id | Operation                   | Name    | Rows  |Bytes| Cost|
--------------------------------------------------------
| 0 | SELECT STATEMENT            |         |     1 |  12 | 3   (0)|
| 1 |  TABLE ACCESS BY INDEX ROWID| TEST_BL |     1 |  12 | 3   (0)|
|*2 |   INDEX UNIQUE SCAN         | TB_PK   |     1 |     | 2   (0)|
---------------------------------------------------------

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
          4  consistent gets
          0  physical reads

The index BLEVEL has gone up to 2 {index height is 3} and now 4 consistent gets are needed to fetch the record.

You may have noticed that the estimated cost of the INDEX_UNIQUE_SCAN is the same as the BLEVEL, which is not really correct. After all, in the first example the cost was 0 and there has to be a read of the index leaf block! The costing makes more sense when it is part of the calculation for scanning an index and then visiting the table for all found records:-

“basic index range scan cost = index blevel + ceil(index selectivity x leaf blocks) + ceil(table selectivity x clustering factor)”

In words this formula means “go down the index to the lead nodes (this is the BLEVEL), scan the number of leaf nodes expected for this index value, then visit the number of table blocks this set of index entries would map to”.

For more information on the formula, I’d plug part of that formula into google (or bing or whatever takes your fancy, search-engine-wise). The original is this page by Richard Foote but there are some good notes by others as well.

There are a lot of references on the web about the cost of accesing an index being the BLEVEL, but remember, if it is a unique access it is the BLEVEL plus one, and oracle seems (in my little tests anyway) to be underestimating the cost by 1. I think this reference to the BLEVEL and the costs might be leading to people (like me, hehe, mistaking the BLEVEL as the actual height of the index.

I was just recently asked the question by a fellow Analysis Services expert (Greg Galloway):

When you have SharePoint automate a data refresh for a PowerPivot workbook, is the new data saved back into the xlsx? So if I download the xlsx onto my laptop the next day, will I see fresh data? Also, if the xlsx is updated by SharePoint during data refresh, is that step subject to the 2GB file limit?

and I thought it made sense if that we share this answer as well.

So the answer is “Yes”.

…

Details you ask? Okay, here are the basic constructs:

Read more…

Disponibilidade 24×7 …sei

ORA-00372: file 28 cannot be modified at this time ORA-01110: data file 28: ‘/u01/app/oracle/admin/mlrepap/db/metalink_02.dbf’

I was just recently asked the question by a fellow Analysis Services expert (Greg Galloway):

When you have SharePoint automate a data refresh for a PowerPivot workbook, is the new data saved back into the xlsx? So if I download the xlsx onto my laptop the next day, will I see fresh data? Also, if the xlsx is updated by SharePoint during data refresh, is that step subject to the 2GB file limit?

and I thought it made sense if that we share this answer as well.

So the answer is “Yes”.

…

Details you ask? Okay, here are the basic constructs:

• When Data Refresh is started, PowerPivot service will reference an existing IMBI database (if it exists) within the Analysis Services Engine Service running within the SharePoint farm
• Or it will pull the database out of the Excel workbook (upon getting the workbook from the SharePoint content database) and attach it to one of the Analysis Services Engine SharePoint Service.  Since it is a service within SharePoint, it makes use of the SharePoint load balancing service (within the farm) to choose which Analysis Services Engine Service to use.  By default, it uses the health-based monitoring to determine on which server to put the database.
• The Data Refresh kicks off and the data is imported into the database running in the Analysis Services Engine SharePoint Service.
• Now that the data is refreshed, the database will be pushed back into the Excel workbook and then saved back to the SharePoint content database.
• Because it is saved in the SharePoint content database, indeed it will be constrained to the 2GB limitation.
• From this point, when you download the file after the data refresh, it will now contain the latest and greatest data.

Later on, I will do blog post based on the high level architecture that may help provide a little better context in all of this as well.

Enjoy!

During my and Dave Wickert’s SQLPASS session (SQLCAT: A Preview to PowerPivot Best Practices), we had shown to the audience how to view and better understand the PowerPivot database file structure within the Excel workbook.  First of all, I’d like to give credit where credit is due – and the thanks really should go to Ashvini Sharma – Power Pivot, Analysis Services, and Integration Services Dev MasterMind.  So while the presentation piece is ours – the actual knowledge and know how is his .

Saying this, what do we mean by all of this?

For starters, recall that an Excel utilizing the PowerPivot add-in actually contains the database within the Excel workbook.  For more information on this, please refer to For Excel PowerPivot, the database is IN the workbook.

But now that you know that the database is IN the workbook, how can you find out why the workbook is so large as my esteemed PowerPivot twitter @VidasM recently inquired?

Well, if you recall from the presentation or blog post, when Excel utilizes PowerPivot, it makes use of the VertiPaq database.  While the database is primarily in-memory, it will have an on-disk structure for it cache against before finally saving itself into the workbook itself (when saved and no longer in use).  To find this temporary cache folder,

• Open up an Excel PowerPivot workbook
• Go to your %TEMP% folder (e.g. c:\Users\MyName\AppData\Local\Temp) and find the IMBI_<GUID> folder.  If there are multiple IMBI_<GUID> folders, find the most recent one
• When you open up this folder, for those of you who work regularly with Analysis Services, this file structure will be very familiar as it is very similar to your native Analysis Services data folder structure.

• Within the [Search Box] (upper right hand corner of your Windows Explorer for those using Windows 7) and type *.* and sort by size.

• By doing so, you can now identify which IDF and dictionary files exist and which files are larger which correspond to the tables and columns you have imported.

While we have yet to develop best practices, the basic principal here is by doing this, you will be able to find the tables and columns that are taking up alot of space within your Excel PowerPivot workbook and can potentially remove any columns that are not required.

Concerning Excel PowerPivot, the question is often asked how is it possible for Excel to now handle tens or hundreds of millions of rows – the way native Excel handles tens or hundreds of thousands of rows. 

How you ask? 

Well, the reason why is because Excel (with the PowerPivot add-in) is using the new In-Memory BI (IMBI) database.  It utilizes a column-based store which has been seen academically and empirically to have some interesting characteristics within BI scenarios…namely, it can calculate many (but not all) BI aggregate very quickly.  Because it is in-memory, it does not utilize disk I/O and performs all of its scans in-memory – i.e. much faster because you do not have disk contention.  As well, it has a great compression algorithm in order to significantly reduce the size of the overall data stored.  By the way, this new database architecture is called VertiPaq for all those interested in the trivia!

So while Vertipaq is really cool, what does this have to do with Excel PowerPivot?  Well, the reason is because the PowerPivot database is actually in the workbook itself.  To prove it, here’s what you can do:

• Take your Excel PowerPivot workbook, which you’ll notice is typically much larger than your typical native Excel workbooks
• Copy the Excel file and rename it using a .zip extension instead of a .xlsx.  Note, the reason why is because .xlsx is actually XML

• When you open up the .zip file, you’ll notice a lot of various folders within it (as noted, it’s all XML).

• Open up the xl folder, and then you’ll notice the customData folder.  When you navigate to it, you’ll notice the item1.data file which contains the bulk of the size of the Excel file

This item1.data file is your PowerPivot database stored right IN the Excel file.

क्या तकलीफ है मुझे समझ नहीं आता. जब से engineering मोल ली है यब से जान की आफत बने बैठे हैं ये exams. अब कल से फिर चालू. दुखी ही कर रखा है.
अब तो inert ही हो गए हैं. हर महीने दे दे के exams अब फर्क पड़ना ही बंद हो गया है.

There is small addition to oradebug unit_test command in 11g R2 .

oradebug unit_test per_session_find_one_waiter

This command has four mandatory arguments and one optional.

First parameter is find_waiters_for which can have two different values : current_sess or all_local_sess .Self descriptive.
Second parameter is wait_event.This is name of the wait event which search is done.Third and fourth parameters are related to time and rusults.

For my simple test I will test this command on event enq TX -row lock contention for my own session.

Session 1:

create table test(a int primary key);

insert into test select rownum from dba_tables where rownum <9;
commit;
update test set a=9 where a=1;
update test set a=10 where a=2;

Session 2:

update test set a=11 where a=3;
update test set a=12 where a=4;

SQL> /

PID SPID                            SID USERNAME                          SERIAL# SQL_HASH_VALUE
———- ———————— ———- —————————— ———- ————–
22 1030                            190 SYS                                  1115     3889092034

Session 1 again:

SQL> /

PID SPID                    SID USERNAME                          SERIAL# SQL_HASH_VALUE
———- ———————— ———- —————————— ———- ————–
21 31360                   156 SYS                                 59123     3889092034

update test set a=25 where a=8;
update test set a=13 where a=3;

Now I got  event to test oradebug command:

oradebug unit_test per_session_find_one_waiter find_waiters_for=current_sess wait_event=”enq: TX – row lock contention” waiter_min_secs_blkd=300 min_results=1

SQL> oradebug unit_test per_session_find_one_waiter find_waiters_for=current_sess wait_event=”enq: TX – row lock contention” waiter_min_secs_blkd=1 min_results=1

Here is output:

WAITERS_FOUND_BEGIN
(inst=1, sid=190, osid=1030) is blocking (inst=1, sid=156) for at least 74 secs
WAITERS_FOUND_END

if I run same command again

SQL> oradebug unit_test per_session_find_one_waiter find_waiters_for=current_sess wait_event=”enq: TX – row lock contention” waiter_min_secs_blkd=1 min_results=1

WAITERS_FOUND_BEGIN
(inst=1, sid=190, osid=1030) is blocking (inst=1, sid=156) for at least 221 secs
WAITERS_FOUND_END

wait time will increase.

Let’s increase waiter_min_secs_blkd=300 and run command:

SQL> oradebug unit_test per_session_find_one_waiter find_waiters_for=current_sess wait_event=”enq: TX – row lock contention” waiter_min_secs_blkd=300 min_results=1
WAITERS_FOUND_BEGIN
(inst=1, sid=190, osid=1030) is blocking (inst=1, sid=156) for at least 301 secs
WAITERS_FOUND_END

and I got output after aprox 301 secs..

Whilst looking for the minimum privileges I needed to execute DBMS_STATS.FLUSH_DATABASE_MONITORING_INFO {it is ANALYZE ANY, by the way} I discovered something about PL/SQL and roles that I did not know. Now, any right I had to claim expertise in PL/SQL expired at least 8 years ago but I asked some friends who ARE still professional PL/SQL experts and they did not know this either.

Privileges granted via Roles to a user are not available to stored PL/SQL created by that user, correct? This is widely known and understood. You have to grant priveleges directly to the user for them to be seen in the PL/SQL packages, functions etc.

Having found that I needed the ANALYZE ANY privilege as I mentioned above, I asked the DBA team to grant my user that privilege on Production and Staging. They did so – via a role. “it won’t work” I said “I run the code via a package, it won’t see the privilege” and proved it by running DBMS_STATS.FLUSH_DATABASE_MONITORING_INFO via a quickly constructed demo. Except I only proved my ignorance, it worked. WHY?

If a package is created such that it is executed with invokers rights then roles are seen. 

This is my test script:

create or replace package test1 is
procedure run_flush;
end test1;
/
--
create or replace package test1 authid current_user is
procedure run_flush is
cursor get_ses_roles is
select role
from session_roles;
begin
  dbms_output.put_line('starting');
  for ses_roles_rec in get_ses_roles loop
    dbms_output.put_line(ses_roles_rec.role);
  end loop;
  dbms_output.put_line('flushing');
  dbms_stats.flush_database_monitoring_info;
  dbms_output.put_line('ending');
end;
begin
  null;
end;
/

I create this package as user MDW.

Now as a privileged user I create a role and grant analyze_any to the role.

MGR>create role mdw_role
Role created.
MGR>grant analyze any to mdw_role;
Grant succeeded.

I’ll just prove that user MDW cannot yet execute the monitoring procedure

MDW> exec dbms_stats.flush_database_monitoring_info
BEGIN dbms_stats.flush_database_monitoring_info; END;
*
ERROR at line 1:
ORA-20000: Insufficient privileges
ORA-06512: at "SYS.DBMS_STATS", line 2148
ORA-06512: at "SYS.DBMS_STATS", line 14135
ORA-06512: at line 1

Now I grant the role

MGR>grant mdw_role to mdw
Grant succeeded.

MDW has to log out and back in again to see the role correctly. Having done this I check for the role and then try to execute the test procedure:

MDW> select * from session_roles

ROLE
------------------------------
CONNECT
MDW_ROLE
2 rows selected.

MDW> exec test1.run_flush
starting
CONNECT
MDW_ROLE
flushing
ending

PL/SQL procedure successfully completed.

You can see that the package sees the roles and it executes the procedure successfully. So, stored PL/SQL can utilise privileges via roles if the packages is created with authid current_user, ie executors rights.

I better admit, as someone else might raise it, that this is not the best demonstration of this feature. I recreated the package with the first line set to:

create or replace package test1 is

ie the default of owners privileges. I now re-execute the call to the package:-

MDW> exec test1.run_flush
starting
flushing
ending

PL/SQL procedure successfully completed.

Note that the roles are no longer seen. However, the DBMS_STATS.FLUSH_DATABASE_MONITORING_INFO still executed correctly.
Why?
Well, because if you look at the package specification of DBMS_STATS you see:-

create or replace package sys.dbms_stats authid current_user is

It makes sense. It is dangerous for system-owned packages to be executing as the owner, ie SYS, as execute access to the package would allow access to SYS privileges.

Which, of course, is why my little proof script executed the flush correctly and I looked a bit silly in front of the DBA team. Oh well.

During a training session today, I was asked about the structure of the Uniquifier, and whether it was a straight identity column. Off the top of my head I couldn’t remember the exact structure, I considered it a 4 byte int, but was not sure whether it acted as a pure identity value or acted in a more complex manner when incrementing, so decided to investigate it tonight.

To start from the beginning, an index uniquifier is the term given to the field that is automatically generated by SQL Server when you create a clustered index, but the index key is not specified as unique. Since each record in the table has to be uniquely identifiable, SQL will automatically assigned a 4 byte field to the row to make it unique, commonly called the ‘Uniquifier’. At this point I am sure English scholars will be frowning, pondering on the nature of the word and whether it qualifies as English; however that the term used so we will run with it.

It is actually quite easy to see this field in action, let’s create a simple table:

CREATE TABLE dbo.unique_test  (  
firstname char(20) NOT NULL,  
surname char(20) NOT NULL  
)  ON [PRIMARY] GO
CREATE CLUSTERED INDEX [ix_test] ON [dbo].[unique_test]
 (
[firstname] ASC
)
ON [PRIMARY]

The clustered index is not unique, by design, so let’s start adding duplicate rows to see the effect:

insert into unique_test values ('John', 'Smith')
go 10

The table now contains 10 rows, each with the same details. This does not cause any undue concern, because each row is actually still unique – the way to show this is using the DBCC INC and DBCC Page commands, I’ve cut the output down since it is so wide.

dbcc ind ('testdb','unique_test',1)
PageFID PagePID     IAMFID IAMPID      PageType
------- ----------- ------ ----------- --------
1       41          NULL   NULL        10      
1       174         1      41          1       

The output shows a data page numbered 174 for my example and the IAM page with an ID of 41. We can crack open the page and view the contents very easily using DBCC Page.

dbcc dbcc traceon(3604)
dbcc page (idtest,1,174,3)

The output is quite large, but in essence, the first record is stored with the following details: 

UNIQUIFIER = [NULL]                 
Slot 0 Column 1 Offset 0x4 Length 20
firstname = John                    
Slot 0 Column 2 Offset 0x18 Length 20

 The second record:

Slot 1 Column 0 Offset 0x33 Length 4
UNIQUIFIER = 1                      
Slot 1 Column 1 Offset 0x4 Length 20
firstname = John                    
Slot 1 Column 2 Offset 0x18 Length 20
surname = Smith  

 The third record:

Slot 2 Column 0 Offset 0x33 Length 4
UNIQUIFIER = 2                      
Slot 2 Column 1 Offset 0x4 Length 20
firstname = John                    
Slot 2 Column 2 Offset 0x18 Length 20
surname = Smith                     

And so forth. The first record’s uniquifier is visible and clearly named within the data page, but set to null. The second copy of the same value receives the uniquifier of one, the third copy receives a 2 etc.  This count is maintained separately for each duplication, so the insert of a new name multiple times will also receive its own counter, beginning at null and working upwards, 1,2,3 etc. So just because the uniquifier is 4 bytes, this does not limit the total number of rows in the table to ~2.1 billion, but does logically limit the total number of duplicates to 2.1 billion. I must confess to not having tested that limit, generating 2.1 billion rows of duplicate data is not trivial and a scrapbook calculation predicts 435 hours of processing on a virtual pc. I suspect the error message it raises when it hits the limit would be interesting.

If we remove all the rows from the table and then add 10 more does the uniquifier reset? Easy to test but the short answer was no, the uniquifier continued to rise, 10 thru 19.

I was a bit suspicious of this since any requirement for the uniquifier to rise / remember what existed before requires it to be stored somewhere – it has to survive a crash after all, but there is no apparent place the current count is stored. If there was, you wouldn’t be storing just 1 value, you would be forced to store a value for each record key that had duplicates. This could run into thousands of separate counters being maintained per clustered key so it just doesn’t make sense that it is stored, it would be a very noticable overhead.

When checking the DBCC Ind for the empty table it insisted it still had a data page, but the only contents of the data page was a single ghost record – a row that has been marked as deleted. The ghost record was the for the ‘John Smith’ with the highest uniquifier before, was this coincidence? The other ghost records had not hung around, so why did this one persist?

I dropped and recreated the table again, inserted 10 rows and then deleted them. Checking DBCC Ind the table still showed a HoBT IAM allocation page for the table and a data page, the data page contained a single ghost record, the one with a Uniquifier of 9 – the highest given out when 10 duplicates were added. Even waiting some considerable time the ghost record was not cleaned up, so it appears that it will not delete it.

If I added another duplicate row, it picked up the next number in the sequence (10) and shortly after the ghost record was removed from the page. Very convenient and not a coincidence at all – the memory of the last uniquifier given out  persists as a ghost record, even if all the duplicates for the table have been removed.  What seems strange is this ghost record hanging about, persisting an entire record, to keep the duplicate count for that key, when no instances of it remain on the table.

It can not possibly do this for every key since the space overhead would become very noticable again, so how does it choose what to persist, the last entry? unfortunately it doesn’t appear that simple at all, after a number of tests it appeared to only be interested in keeping the ghost entry for the record that had the highest key value, so alphabetically, the one closed to ‘Z’ for my example.

Conclusion? On the evidence, whilst the other ghost records still persist for a short time, even deleting and then adding more duplicates can see the number continue from where it left of, but given a short time for the ghost records to be removed the uniquifier will restart the sequence back at Null,1,2 etc. Except in the case of the highest entry from the index perspective, that ghost record decides to stick around until there is another entry using the same key, continuing the sequence, at which point the ghost record then finally disappears.

I can not think of any sensible reason why it would do this, can you?

Overall, the uniquifier is a cost overhead of not having a unique index, and at a cost of 4 bytes, an int identity column makes a lot of sense - for all purposes it acts the same and serves the same purpose but in a far more visible manner – so it really does not make much sense to rely on the uniquifier provided for you, take control and create your own.

After decoding the DateTime2 internal structure I thought I would take a quick look at the DateTimeOffset structure, since it should not present too many difficulties and post it up quickly, but was surprised at the initial result of a test. It follows the same basic premise that the time portion is followed by the date portion and the time / date is based on a day count from the epoch time of 0001/01/01 and the time the number of intervals since midnight, where the interval is defined by the accuracy.

I was expecting the datetime offset value itself to occupy the additional 2 bytes quoted, and not affect the other values. Bad assumption, as soon as I cracked open a few examples I could immediately see that setting the offset also alters the underlying time / date component values as well.

Using the same comparison methods as before the underlying time value is clearly being adjusted:

'0001/01/01 00:00:00 -0:00' => 0x0700000000000000000000
'0001/01/01 00:00:00 -12:00' => 0x0700E034956400000030FD

So, even though an offset is being stored, the underlying time is also being altered to match, and the internal storage is using UTC as the reference point. This makes sense as the most valid reference that you could use.

'0001-01-01 12:00:00.0000000 +00:00' => 0x0700E03495640000000000
'0001-01-01 00:00:00.0000000 -12:00' => 0x0700E034956400000030FD

The same time / date is generated for the two values, but the last two bytes hold the offset and have stored the offset used when the date was initially stored. The underlying storage of the time though is clearly identical in both,  so UTC is the common ground they each get stored again.

The final 2 bytes for the offset are pretty easy to decode, since the pattern is the same as before with a slight twist. The offset time records the number of minutes for the offset in hex, with the first byte of the two being the least significant as before with the time, so you end up reading the two bytes left-to-right and then decode that byte right-to-left.

The twist is that for positive offsets, the value increments 1,2,3, in hex as appropriate, but for negative values, it starts decrementing by considering -1 equal to ‘FFFF’, I’ve split the hex output into the individual components by adding some spaces to make it easier to read. (Accuracy Code, Time Value, Date Value, Offset Used)

'2001-01-01 12:00:00.0000000 +00:01' => 0x07   009A717164   75250B  0100
'2001-01-01 12:00:00.0000000 +00:00' => 0x07   00E0349564   75250B  0000
'2001-01-01 12:00:00.0000000 -00:01' => 0x07   0026F8B864   75250B  FFFF

Since the offsets supported at only +14 hours to -14 hours, there is no risk of the two ranges overlapping. When I think about this a bit more, it is acting as a signed number, -1 being 11111111111 etc. So the 2 bytes at the end is a signed int of the number of minutes offset.

There are a number of time zones in the world that do not occur at exact hourly intervals from UTC, some are on the half hour mark such as Caracas (-4:30) or Delhi (+5:30) to name a few, whilst Kathmandu (+5:45) is even more specific.  In theory the format allows offsets specified to even greater levels of distinction, although I am not sure as to why you would wish to use it. Do you really want an offset of +3:17 minutes? That is potentially scary to consider that as a valid input to the value.

That has made me wonder as to why the accuracy was set so high, when in reality 15 minute intervals would of been sufficient, with a -14 to +14 range, that is 113 different values inclusively, which could be accomodated within a single byte.

So why spend 2 bytes on the format, when 1 was enough? Was it to just be compatible to the ISO format in some way that required it? Not sure.

Managed Threading

Threading Objects and Features

A process is a collection of virtual memory space, code, data, and system resources. A thread is code that is to be serially executed within a process. A processor executes threads, not processes, so each application has at least one process, and a process always has at least one thread of execution, known as the primary thread. A process can have multiple threads in addition to the primary thread.

Processes communicate with one another through messages, using Microsoft’s Remote Procedure Call (RPC) technology to pass information to one another. There is no difference to the caller between a call coming from a process on a remote machine and a call coming from another process on the same machine.

When a thread begins to execute, it continues until it is killed or until it is interrupted by a thread with higher priority (by a user action or the kernel’s thread scheduler). Each thread can run separate sections of code, or multiple threads can execute the same section of code. Threads executing the same block of code maintain separate stacks. Each thread in a process shares that process’s global variables and resources.

Threads are the basic unit to which an operating system allocates processor time, and more than one thread can be executing code inside that process. Each thread maintains

• exception handlers,
• a scheduling priority, and
• a set of structures the system uses to save the thread context until it is scheduled.

The thread context includes all the information the thread needs to seamlessly resume execution, including

• the thread’s set of CPU registers and
• stack, in the address space of the thread’s host process.

The .NET Framework further subdivides an operating system process into lightweight managed subprocesses, called application domains, represented by System..::.AppDomain. One or more managed threads (represented by System.Threading..::.Thread) can run in one or any number of application domains within the same managed process. Although each application domain is started with a single thread, code in that application domain can create additional application domains and additional threads. The result is that a managed thread can move freely between application domains inside the same managed process; you might have only one thread moving among several application domains.

If your application uses only one thread of execution, you can combine asynchronous programming with .NET Framework remoting or XML Web services created using ASP.NET to use the processing time of other computers in addition to that of your own to increase responsiveness to the user and decrease the data processing time of your application.

Disadvantages of Multiple Threads

It is recommended that you use as few threads as possible, thereby minimizing the use of operating-system resources and improving performance. Threading also has resource requirements and potential conflicts to be considered when designing your application. The resource requirements are as follows:

• The system consumes memory for the context information required by processes, AppDomain objects, and threads. Therefore, the number of processes, AppDomain objects, and threads that can be created is limited by available memory.
• Keeping track of a large number of threads consumes significant processor time. If there are too many threads, most of them will not make significant progress. If most of the current threads are in one process, threads in other processes are scheduled less frequently.
• Controlling code execution with many threads is complex, and can be a source of many bugs.
• Destroying threads requires knowing what could happen and handling those issues.

Resources that require synchronization include:

• System resources (such as communications ports).
• Resources shared by multiple processes (such as file handles).
• The resources of a single application domain (such as global, static, and instance fields) accessed by multiple threads.

Synchronizing Data for Multithreading

When multiple threads can make calls to the properties and methods of a single object, it is critical that those calls be synchronized. Otherwise one thread might interrupt what another thread is doing, and the object could be left in an invalid state. A class whose members are protected from such interruptions is called thread-safe.

The Common Language Infrastructure provides several strategies to synchronize access to instance and static members:

• Synchronized code regions. You can use the Monitor class or compiler support for this class to synchronize only the code block that needs it, improving performance.
• Manual synchronization.You can use the synchronization objects provided by the .NET Framework class library. See Overview of Synchronization Primitives, which includes a discussion of the Monitor class.
• Synchronized contexts. You can use the SynchronizationAttribute to enable simple, automatic synchronization for ContextBoundObject objects.
• Synchronized property. A few classes, such as Hashtable and Queue, provide a Synchronized property that returns a thread-safe wrapper for an instance of the class. See Collections and Synchronization (Thread Safety).

The common language runtime provides a thread model in which classes fall into a number of categories that can be synchronized in a variety of different ways depending on the requirements. The following table shows what synchronization support is provided for fields and methods with a given synchronization category.

Category	Global fields	Static fields	Static methods	Instance fields	Instance methods	Specific code blocks
No Synchronization	No	No	No	No	No	No
Synchronized Context	No	No	No	Yes	Yes	No
Synchronized Code Regions	No	No	Only if marked	No	Only if marked	Only if marked
Manual Synchronization	Manual	Manual	Manual	Manual	Manual	Manual

Threading and Application Design

In general, using the ThreadPool class is the easiest way to handle multiple threads for relatively short tasks that will not block other threads and when you do not expect any particular scheduling of the tasks. However, there are a number of reasons to create your own threads:

• If you need a task to have a particular priority.
• If you have a task that might run a long time (and therefore block other tasks).
• If you need to place threads into a single-threaded apartment (all ThreadPool threads are in the multithreaded apartment).
• If you need a stable identity associated with the thread. For example, you should use a dedicated thread to abort that thread, suspend it, or discover it by name.
• If you need to run background threads that interact with the user interface, the .NET Framework version 2.0 provides a BackgroundWorker component that communicates using events, with cross-thread marshaling to the user-interface thread.

Apartment Threading (Single Threaded Apartment)

Apartment threaded means there are multiple threads within the application. In single threaded apartment (STA) each thread is isolated in a separate apartment underneath the process. The process can have any number of apartments that share data through a proxy. The application defines when and for how long the thread in each apartment should execute. All requests are serialized through the Windows message queue such that only a single apartment is accessed at a time and thus only a single thread will be executing at any one time. STA is the threading model that most Visual Basic developers are familiar with because this is the threading model available to VB applications prior to VB.NET. You can think of it like an apartment building full of a row of one room apartments that are accessible one at a time through a single hallway. The advantage this provides over single threaded is that multiple commands can be issued at one time instead of just a single command, but the commands are still sequentially executed.

Free Threading (Multi Threaded Apartment)

Free threaded applications were limited to programming languages such as C++ until the release of Microsoft .NET. The free threaded/Multi Threaded Apartment (MTA) model has a single apartment created underneath the process rather than multiple apartments. This single apartment holds multiple threads rather than just a single thread. No message queue is required because all of the threads are a part of the same apartment and can share data without a proxy. You can think of it like a building with multiple rooms that are all accessible once you are inside the building. These applications typically execute faster than single threaded and STA because there is less system overhead and can be optimized to eliminate system idle time.

These types of applications are more complex to program. The developer must provide thread synchronization as part of the code to ensure that threads do not simultaneously access the same resources. A condition known as a race condition can occur when a thread accesses a shared resource and modifies the resource to an invalid state and then another thread accesses the shared resource and uses it in the invalid state before the other thread can return the resource to a valid state. Therefore it is necessary to place a lock on a resource to prevent other threads from accessing the resource until the lock has been removed. However, this can lead to a deadlock situation where two threads are competing for resources and neither can proceed.

The only way to avoid situations like these is through good design and testing.

Rules for single-threaded apartments are simple, but it is important to follow them carefully:

• Every object should live on only one thread (within a single-threaded apartment).
• Initialize the COM library for each thread.
• Marshal all pointers to objects when passing them between apartments.
• Each single-threaded apartment must have a message loop to handle calls from other processes and apartments within the same process. Single-threaded apartments without objects (client only) also need a message loop to dispatch the broadcast messages that some applications use.
• DLL-based or in-process objects do not call the COM initialization functions; instead, they register their threading model with the ThreadingModel named-value under the InprocServer32 key in the registry. Apartment-aware objects must also write DLL entry points carefully. There are special considerations that apply to threading in-process servers. For more information, see In-Process Server Threading Issues.

Following are some important considerations regarding synchronization for multithreaded apartments:

• COM provides call synchronization for single-threaded apartments only.
• Multithreaded apartments do not receive calls while making calls (on the same thread).
• Multithreaded apartments cannot make input-synchronized calls.
• Asynchronous calls are converted to synchronous calls in multithreaded apartments.
• The message filter is not called for any thread in a multithreaded apartment.

Important Note:

Do not lock the type — that is, typeof(MyType) in C#, GetType(MyType) in Visual Basic, or MyType::typeid in C++ — in order to protect static methods (Shared methods in Visual Basic). Use a private static object instead. Similarly, do not use this in C# (Me in Visual Basic) to lock instance methods. Use a private object instead. A class or instance can be locked by code other than your own, potentially causing deadlocks or performance problems.

Synchronized Context

You can use the SynchronizationAttribute on any ContextBoundObject to synchronize all instance methods and fields. All objects in the same context domain share the same lock. Multiple threads are allowed to access the methods and fields, but only a single thread is allowed at any one time.

Setting Up for a Background Operation

To set up for a background operation, add an event handler for the DoWork event. Call your time-consuming operation in this event handler.To start the operation, call RunWorkerAsync. To receive notifications of progress updates, handle the ProgressChanged event. To receive a notification when the operation is completed, handle the RunWorkerCompleted event.The methods that handle the ProgressChanged and RunWorkerCompleted events can access the application’s user interface, because those events are raised on the thread that called the RunWorkerAsync method. However, the DoWork event handler cannot work with any user-interface objects because it runs on the background thread.

Safe Points

Multithreaded Applications.

You can provide parameters and return values to thread-pool threads by using the optional ByVal state-object variable of the QueueUserWorkItem method. Thread-timer threads also support a state object for this purpose. For information on thread pooling and thread timers, see Thread Pooling and Thread Timers.

Sub DoWork()
    ' Queue a task
    System.Threading.ThreadPool.QueueUserWorkItem( _
        New System.Threading.WaitCallback(AddressOf SomeLongTask))
    ' Queue another task
    System.Threading.ThreadPool.QueueUserWorkItem( _
        New System.Threading.WaitCallback(AddressOf AnotherLongTask))
End Sub
Sub SomeLongTask(ByVal state As Object)
    ' Insert code to perform a long task.
End Sub
Sub AnotherLongTask(ByVal state As Object)
    ' Insert code to perform another long task.
End Sub

Application Domains and Threads.

Events (C#).

Events can be used to synchronize threads.

Delegate is a creatable type. Programming with delegates requires creating delegate objects from delegate types.

Delegates seamlessly support binding a notification source to multiple handler methods through a feature known as multicasting.

Delegate binding, see Delegates in the Common Type System and CreateDelegate(Type,Object,MethodInfo,Boolean).

Delegates are referred to as

• multicast, or
• combinable,

because a delegate can invoke one or more methods and can be used in combining operations.A delegate can represent a

• static method or
• an instance method.

When the delegate represents an instance method, the delegate stores not only a reference to the method’s entry point, but also a reference to the class instance. Unlike function pointers,

• delegates are object oriented and
• type safe.

How to: Subscribe to and Unsubscribe from Events (C# Programming Guide)

The line of code that is needed to subscribe to the event is also automatically generated in the InitializeComponent method in the Form1.Designer.cs file in your project. It looks like this:

  this.Load += new System.EventHandler(this.Form1_Load);

To subscribe to events programmatically

1. Define an event handler method whose signature matches the delegate signature for the event. For example, if the event is based on the EventHandler delegate type, then the following code represents the method stub:

     void HandleCustomEvent(object sender, CustomEventArgs a)
     {
        // Do something useful here.
     }

2. Use the addition assignment operator (+=) to attach your event handler to the event. In the following example, assume that an object named publisher has an event named RaiseCustomEvent. Note that the subscriber class needs a reference to the publisher class in order to subscribe to its events.

   publisher.RaiseCustomEvent += HandleCustomEvent;

Note that the above syntax is new in C# 2.0. It is exactly equivalent to the C# 1.0 syntax in which the encapsulating delegate must be explicitly created using the new keyword:

publisher.RaiseCustomEvent += new CustomEventHandler(HandleCustomEvent);

To subscribe to events by using an anonymous method

• Use the addition assignment operator (+=) to attach your anonymous method to the event. In the following example, assume that an object named publisher has an event named RaiseCustomEvent and that a CustomEventArgs class has also been defined to carry some kind of specialized event information. Note that the subscriber class needs a reference to publisher in order to subscribe to its events.

    publisher.RaiseCustomEvent += delegate(object o, CustomEventArgs e)
    {
      string s = o.ToString() + " " + e.ToString();
      Console.WriteLine(s);
    };

It is important to note that you cannot easily unsubscribe from an event if you used an anonymous method to subscribe to it. To unsubscribe in this scenario, go back to the code where you subscribe to the event, store the anonymous method in a delegate variable, and then add the delegate to the event.

When an event has multiple subscribers, the event handlers are invoked synchronously when an event is raised. To invoke events asynchronously, see Calling Synchronous Methods Asynchronously.

four common ways to use BeginInvoke and EndInvoke to make asynchronous calls. After calling BeginInvoke you can do the following:

• Do some work and then call EndInvoke to block until the call completes.
• Obtain a WaitHandle using the System.IAsyncResult.AsyncWaitHandle property, use its WaitOne method to block execution until the WaitHandle is signaled, and then call EndInvoke.
• Poll the IAsyncResult returned by BeginInvoke to determine when the asynchronous call has completed, and then call EndInvoke.
• Pass a delegate for a callback method to BeginInvoke. The method is executed on a ThreadPool thread when the asynchronous call completes. The callback method calls EndInvoke.

  Important
  Always call EndInvoke to complete your asynchronous call.

you should know that events in the .NET Framework are layered on top of delegates. When you use an event-driven application framework such as Windows® Forms or ASP.NET, your knowledge of delegates will make you a much stronger developer.

Delegates provide the primary means in .NET for executing a method on a secondary thread in an asynchronous fashion. Therefore, delegates open the door to multithreading.

the compiler generates a class definition for each delegate type. The class that is generated for each delegate type is a creatable class that inherits from the System.Multicast delegate.

compiler has also added a public constructor and three public methods named

• Invoke,
• BeginInvoke, and
• EndInvoke.

Asynchronous Programming Design Patterns

Asynchronous operations are typically used to perform tasks that might take a long time to complete, such as opening large files, connecting to remote computers, or querying a database. An asynchronous operation executes in a thread separate from the main application thread. When an application calls methods to perform an operation asynchronously, the application can continue executing while the asynchronous method performs its task.

Asynchronous programming is a feature supported by many areas of the .NET Framework, including:

• File IO, Stream IO, Socket IO.
• Networking.
• Remoting channels (HTTP, TCP) and proxies.
• XML Web services created using ASP.NET.
• ASP.NET Web Forms.
• Message Queuing using the MessageQueue class.

The .NET Framework provides two design patterns for asynchronous operations:

• Asynchronous operations that use IAsyncResult objects.
• Asynchronous operations that use events. Event-based Asynchronous Pattern Overview.

For relatively simple multithreaded applications, the BackgroundWorker component provides a straightforward solution. For more sophisticated asynchronous applications, consider implementing a class that adheres to the Event-based Asynchronous Pattern.

Asynchronous Programming Overview

Discusses the IAsyncResult based asynchronous design pattern, which provides a flexible programming model to deal with asynchronous operations.

The correct choice depends on whether the application has instructions that can execute while the operation completes. If an application cannot perform any additional work until it receives the results of the asynchronous operation, the application must block until the results are available. To block until an asynchronous operation completes, you can use one of the following approaches:

• Call EndOperationName from the application’s main thread, blocking application execution until the operation is complete. For an example that illustrates this technique, see Blocking Application Execution by Ending an Asynchronous Operation.
• Use the AsyncWaitHandle to block application execution until one or more operations are complete. For an example that illustrates this technique, see Blocking Application Execution Using an AsyncWaitHandle.

Applications that do not need to block while the asynchronous operation completes can use one of the following approaches:

• Poll for operation completion status by checking the IsCompleted property periodically and calling EndOperationName when the operation is complete. For an example that illustrates this technique, see Polling for the Status of an Asynchronous Operation.
• Use an AsyncCallback delegate to specify a method to be invoked when the operation is complete. For an example that illustrates this technique, see Using an AsyncCallback Delegate to End an Asynchronous Operation.

Creating Event Handlers in Windows Forms

• How to: Subscribe to and Unsubscribe from Events (C# Programming Guide)
• How to: Publish Events that Conform to .NET Framework Guidelines (C# Programming Guide)
• How to: Raise Base Class Events in Derived Classes (C# Programming Guide)
• How to: Implement Interface Events (C# Programming Guide)
• Thread Synchronization (C# Programming Guide)
• How to: Use a Dictionary to Store Event Instances (C# Programming Guide)
• Events Sample
• Event Design

Multithreading in Components

Managed Threading Best Practices

Implementing the Event-based Asynchronous Pattern

Describes the standardized way to package a class that has asynchronous features.

features of the Event-based Asynchronous Pattern discussed in this topic.

• Opportunities for Implementing the Event-based Asynchronous Pattern
• Naming Asynchronous Methods
• Optionally Support Cancellation
• Optionally Support the IsBusy Property
• Optionally Provide Support for Progress Reporting
• Optionally Provide Support for Returning Incremental Results
• Handling Out and Ref Parameters in Methods

Deciding When to Implement the Event-based Asynchronous Pattern

Describes how to determine when you should choose to implement the Event-based Asynchronous Pattern instead of the IAsyncResult pattern.

Event-based Asynchronous Pattern Technology Sample

Shows how to use the Event-based Asynchronous Pattern to perform common asynchronous operations.

Events and Delegates

How to: Run an Operation in the Background

use the BackgroundWorker class to run the operation on another thread.

Robust Programming

Caution
When using multithreading of any sort, you potentially expose yourself to very serious and complex bugs. Consult the Managed Threading Best Practices before implementing any solution that uses multithreading.

Deadlocks

A deadlock occurs when each of two threads tries to lock a resource the other has already locked. Neither thread can make any further progress.

Many methods of the managed threading classes provide time-outs to help you detect deadlocks. For example, the following code attempts to acquire a lock on the current instance. If the lock is not obtained in 300 milliseconds, Monitor.TryEnter returns false.

Race Conditions

A race condition is a bug that occurs when the outcome of a program depends on which of two or more threads reaches a particular block of code first. Running the program many times produces different results, and the result of any given run cannot be predicted.

A simple example of a race condition is incrementing a field. Suppose a class has a private static field (Shared in Visual Basic) that is incremented every time an instance of the class is created, using code such as objCt++; (C#) or objCt += 1 (Visual Basic). This operation requires loading the value from objCt into a register, incrementing the value, and storing it in objCt.

In a multithreaded application, a thread that has loaded and incremented the value might be preempted by another thread which performs all three steps; when the first thread resumes execution and stores its value, it overwrites objCt without taking into account the fact that the value has changed in the interim.

This particular race condition is easily avoided by using methods of the Interlocked class, such as Interlocked.Increment. To read about other techniques for synchronizing data among multiple threads, see Synchronizing Data for Multithreading.

Race conditions can also occur when you synchronize the activities of multiple threads. Whenever you write a line of code, you must consider what might happen if a thread were preempted before executing the line (or before any of the individual machine instructions that make up the line), and another thread overtook it.

Overview of Synchronization Primitives

The .NET Framework provides a range of synchronization primitives for controlling the interactions of threads and avoiding race conditions. These can be roughly divided into three categories:

• locking,
• signaling, and
• interlocked operations.

The categories are not tidy and clearly defined: some synchronization mechanisms have characteristics of multiple categories; events that release a single thread at a time are functionally like locks; the release of any lock can be thought of as a signal; and interlocked operations can be used to construct locks. However, the categories are still useful.

Signaling

The simplest way to wait for a signal from another thread is to call the Join method, which blocks until the other thread completes. Join has two overloads that allow the blocked thread to break out of the wait after a specified interval has elapsed.

Wait handles provide a much richer set of waiting and signaling capabilities.

Wait Handles

Wait handles derive from the WaitHandle class, which in turn derives from MarshalByRefObject. Thus, wait handles can be used to synchronize the activities of threads across application domain boundaries.

Threads block on wait handles by calling the instance method WaitOne or one of the static methods WaitAll, WaitAny, or SignalAndWait. How they are released depends on which method was called, and on the kind of wait handles.

For a conceptual overview, see Wait Handles. Also EventWaitHandle, AutoResetEvent, and ManualResetEvent.

Interlocked Operations

Interlocked operations are simple atomic operations performed on a memory location by static methods of the Interlocked class. Those atomic operations include addition, increment and decrement, exchange, conditional exchange depending on a comparison, and read operations for 64-bit values on 32-bit platforms.

Note:
The guarantee of atomicity is limited to individual operations; when multiple operations must be performed as a unit, a more coarse-grained synchronization mechanism must be used.

Although none of these operations are locks or signals, they can be used to construct locks and signals. Because they are native to the Windows operating system, interlocked operations are extremely fast.

Interlocked operations can be used with volatile memory guarantees to write applications which exhibit powerful non-blocking concurrency, however they require sophisticated, low level programming, and for most purposes simple locks are a better choice.

For a conceptual overview, see Interlocked Operations.

SynchronizationAttribute

When this attribute is applied to an object, only one thread can be executing in all contexts that share an instance of this property. This is done by contributing sinks that intercept and serialize incoming calls for the respective contexts. If the property is marked for reentry, then callouts are intercepted too. The callout interception allows other waiting threads to enter the synchronization domain for maximal throughput.

Note:

There are two classes named SynchronizationAttribute : one in the System.Runtime.Remoting.Contexts namespace, and the other in the System.EnterpriseServices namespace. The System.EnterpriseServices..::.SynchronizationAttribute class supports only synchronous calls, and can be used only with serviced components. (For more information on serviced components, see [<topic://cpconservicedcomponentoverview>].) The System.Runtime.Remoting.Contexts..::.SynchronizationAttribute supports both synchronous and asynchronous calls, and can be used only with context bound objects. (For more information on context bound objects, see the ContextBoundObject class.)

Recommendations for Class Libraries

Consider the following guidelines when designing class libraries for multithreading:

• Avoid the need for synchronization, if possible. This is especially true for heavily used code. For example, an algorithm might be adjusted to tolerate a race condition rather than eliminate it. Unnecessary synchronization decreases performance and creates the possibility of deadlocks and race conditions.
• Make static data (Shared in Visual Basic) thread safe by default.
• Do not make instance data thread safe by default. Adding locks to create thread-safe code decreases performance, increases lock contention, and creates the possibility for deadlocks to occur. In common application models, only one thread at a time executes user code, which minimizes the need for thread safety. For this reason, the .NET Framework class libraries are not thread safe by default.
• Avoid providing static methods that alter static state. In common server scenarios, static state is shared across requests, which means multiple threads can execute that code at the same time. This opens up the possibility of threading bugs. Consider using a design pattern that encapsulates data into instances that are not shared across requests. Furthermore, if static data are synchronized, calls between static methods that alter state can result in deadlocks or redundant synchronization, adversely affecting performance.

General Recommendations

Consider the following guidelines when using multiple threads:

• Don’t use Thread.Abort to terminate other threads. Calling Abort on another thread is akin to throwing an exception on that thread, without knowing what point that thread has reached in its processing.
• Don’t use Thread.Suspend and Thread.Resume to synchronize the activities of multiple threads. Do use
• • Mutex,
  • ManualResetEvent,
  • AutoResetEvent,
  • Monitor.
• Don’t control the execution of worker threads from your main program (using events, for example). Instead, design your program so that worker threads are responsible for waiting until work is available, executing it, and notifying other parts of your program when finished. If your worker threads do not block, consider using thread pool threads. Monitor.PulseAll is useful in situations where worker threads block.
• The Birth of the Thread Pool
  Capability 1: Calling a Method Asynchronously
  Capability 2: Calling a Method at Timed Intervals
  Capability 3: Calling a Method When a Single Kernel Object Becomes Signaled
• Don’t use types as lock objects. That is, avoid code such as lock(typeof(X)) in C# or SyncLock(GetType(X)) in Visual Basic, or the use of System.Threading.Monitor.Enter(System.Object) with Type objects. For a given type, there is only one instance of System.Type per application domain. If the type you take a lock on is public, code other than your own can take locks on it, leading to deadlocks. For additional issues, see Reliability Best Practices.
• Use caution when locking on instances, for example lock(this) in C# or SyncLock(Me) in Visual Basic. If other code in your application, external to the type, takes a lock on the object, deadlocks could occur.
• Do ensure that a thread that has entered a monitor always leaves that monitor, even if an exception occurs while the thread is in the monitor. The C# lock statement and the Visual Basic SyncLock statement provide this behavior automatically, employing a finally block to ensure that Monitor.Exit is called. If you cannot ensure that Exit will be called, consider changing your design to use Mutex. A mutex is automatically released when the thread that currently owns it terminates.
• Do use multiple threads for tasks that require different resources, and avoid assigning multiple threads to a single resource. For example, any task involving I/O benefits from having its own thread, because that thread will block during I/O operations and thus allow other threads to execute. User input is another resource that benefits from a dedicated thread. On a single-processor computer, a task that involves intensive computation coexists with user input and with tasks that involve I/O, but multiple computation-intensive tasks contend with each other.
• Consider using methods of the Interlocked class for simple state changes, instead of using the lock statement (SyncLock in Visual Basic). The lock statement is a good general-purpose tool, but the Interlocked class provides better performance for updates that must be atomic. Internally, it executes a single lock prefix if there is no contention. In code reviews, watch for code like that shown in the following examples. In the first example, a state variable is incremented:

Performance Considerations with Locks, Threads, Semaphores

You can improve performance by using the Increment method instead of the lock statement, as follows:

C#

System.Threading.Interlocked.Increment(myField);

In the second example, a reference type variable is updated only if it is a null reference (Nothing in Visual Basic).

C#

if (x == null)
{
    lock (lockObject)
    {
        if (x == null)
        {
            x = y;
        }
    }
}

Performance can be improved by using the CompareExchange method instead, as follows:

C#

System.Threading.Interlocked.CompareExchange(ref x, y, null);

The methods of this class help protect against errors that can occur when the scheduler switches contexts while a thread is updating a variable that can be accessed by other threads, or when two threads are executing concurrently on separate processors. The members of this class do not throw exceptions.The Increment and Decrement methods increment or decrement a variable and store the resulting value in a single operation. On most computers, incrementing a variable is not an atomic operation, requiring the following steps:

1. Load a value from an instance variable into a register.
2. Increment or decrement the value.
3. Store the value in the instance variable.

If you do not use Increment and Decrement, a thread can be preempted after executing the first two steps. Another thread can then execute all three steps. When the first thread resumes execution, it overwrites the value in the instance variable, and the effect of the increment or decrement performed by the second thread is lost.

The Exchange method atomically exchanges the values of the specified variables. The CompareExchange method combines two operations: comparing two values and storing a third value in one of the variables, based on the outcome of the comparison. The compare and exchange operations are performed as an atomic operation.

//A simple method that denies reentrancy.
        static bool UseResource()
        {
            //0 indicates that the method is not in use.
            if(0 == Interlocked.Exchange(ref usingResource, 1))
            {
                Console.WriteLine("{0} acquired the lock", Thread.CurrentThread.Name);

                //Code to access a resource that is not thread safe would go here.

                //Simulate some work
                Thread.Sleep(500);

                Console.WriteLine("{0} exiting lock", Thread.CurrentThread.Name);

                //Release the lock
                Interlocked.Exchange(ref usingResource, 0);
                return true;
            }
            else
            {
                Console.WriteLine("   {0} was denied the lock", Thread.CurrentThread.Name);
                return false;
            }
        }

Source Code Examples

• Part 1 – Creating a thread in Windows Forms -  Some basic ideas about firing a background and how to synchronize it with the UI.
• Part 2 – Using BackgroundWorker -  the .NET 2.0 way using a BackgroundWorker component

• Part 3 – Automatic Marshalling – creating your own component with auto marshalling

When Not to Use Thread Pool Threads

There are several scenarios in which it is appropriate to create and manage your own threads instead of using thread pool threads:

• You require a foreground thread.
• You require a thread to have a particular priority.
• You have tasks that cause the thread to block for long periods of time. The thread pool has a maximum number of threads, so a large number of blocked thread pool threads might prevent tasks from starting.
• You need to place threads into a single-threaded apartment. All ThreadPool threads are in the multithreaded apartment.
• You need to have a stable identity associated with the thread, or to dedicate a thread to a task.

BackgroundWorker Component Overview

Describes the BackgroundWorker component, which gives you the ability to execute time-consuming operations asynchronously (“in the background”), on a thread different from your application’s main UI thread.

There are many commonly performed operations that can take a long time to execute. For example:

• Image downloads
• Web service invocations
• File downloads and uploads (including for peer-to-peer applications)
• Complex local computations
• Database transactions
• Local disk access, given its slow speed relative to memory access

System.MarshalByRefObject
System.ComponentModel.Component
System.ComponentModel.BackgroundWorker

BackgroundWorker

Describes this class and has links to all its members.

To execute a time-consuming operation in the background, create a BackgroundWorker and listen for events that report the progress of your operation and signal when your operation is finished. You can create the BackgroundWorker programmatically or you can drag it onto your form from the Components tab of the Toolbox. If you create the BackgroundWorker in the Windows Forms Designer, it will appear in the Component Tray, and its properties will be displayed in the Properties window.

To set up for a background operation, add an event handler for the DoWork event. Call your time-consuming operation in this event handler. To start the operation, call RunWorkerAsync. To receive notifications of progress updates, handle the ProgressChanged event. To receive a notification when the operation is completed, handle the RunWorkerCompleted event.

Note:
You must be careful not to manipulate any user-interface objects in your DoWork event handler. Instead, communicate to the user interface through the ProgressChanged and RunWorkerCompleted events.BackgroundWorker events are not marshaled across AppDomain boundaries. Do not use a BackgroundWorker component to perform multithreaded operations in more than one AppDomain.

If your background operation requires a parameter, call RunWorkerAsync with your parameter. Inside the DoWork event handler, you can extract the parameter from the DoWorkEventArgs.Argument property.

System.Threading..::.ThreadPool

Remarks

Note:

The HostProtectionAttribute attribute applied to this type or member has the following Resources property value: Synchronization | ExternalThreading. The HostProtectionAttribute does not affect desktop applications (which are typically started by double-clicking an icon, typing a command, or entering a URL in a browser). For more information, see the HostProtectionAttribute class or SQL Server Programming and Host Protection Attributes.

Many applications create threads that spend a great deal of time in the sleeping state, waiting for an event to occur. Other threads might enter a sleeping state only to be awakened periodically to poll for a change or update status information. Thread pooling enables you to use threads more efficiently by providing your application with a pool of worker threads that are managed by the system. One thread monitors the status of several wait operations queued to the thread pool. When a wait operation completes, a worker thread from the thread pool executes the corresponding callback function.

Note:
The threads in the managed thread pool are background threads. That is, their IsBackground properties are true. This means that a ThreadPool thread will not keep an application running after all foreground threads have exited.

You can also queue work items that are not related to a wait operation to the thread pool. To request that a work item be handled by a thread in the thread pool, call the QueueUserWorkItem method. This method takes as a parameter a reference to the method or delegate that will be called by the thread selected from the thread pool. There is no way to cancel a work item after it has been queued.

Timer-queue timers and registered wait operations also use the thread pool. Their callback functions are queued to the thread pool.

There is one thread pool per process. The thread pool has a default size of 250 worker threads per available processor, and 1000 I/O completion threads. The number of threads in the thread pool can be changed by using the SetMaxThreads method. Each thread uses the default stack size and runs at the default priority.

The thread pool maintains a minimum number of idle threads. For worker threads, the default value of this minimum is the number of processors. The GetMinThreads method obtains the minimum numbers of idle worker and I/O completion threads.

When all thread pool threads have been assigned to tasks, the thread pool does not immediately begin creating new idle threads. To avoid unnecessarily allocating stack space for threads, it creates new idle threads at intervals. The interval is currently half a second, although it could change in future versions of the .NET Framework.

If an application is subject to bursts of activity in which large numbers of thread pool tasks are queued, use the SetMinThreads method to increase the minimum number of idle threads. Otherwise, the built-in delay in creating new idle threads could cause a bottleneck.

Note:
Unnecessarily increasing the number of idle threads can also cause performance problems. Stack space must be allocated for each thread. If too many tasks start at the same time, all of them might appear to be slow. Finding the right balance is a performance-tuning issue.

When the thread pool reuses a thread, it does not clear the data in thread local storage or in fields that are marked with the ThreadStaticAttribute attribute. Therefore, data that is placed in thread local storage by one method can be exposed to any other method that is executed by the same thread pool thread. A method that accesses a field that is marked with the ThreadStaticAttribute attribute could encounter different data depending on which thread pool thread executes it.

Starting with the .NET Framework version 2.0 Service Pack 1, the throughput of the thread pool is significantly improved for applications that make heavy use of small thread pool tasks. These applications will see improvements in three areas:

• queuing tasks,
• dispatching thread pool threads, and
• dispatching I/O completion threads.

To use this functionality, your application should target the .NET Framework version 3.5. For more information, see .NET Framework 3.5 Architecture.

More Samples, .NET Snippets

use the Invoke Method of the ISynchronizeInvoke Interface:

CODE

#region UpdateStatus
private static void UpdateStatus(string msg, string status)
{
//create our Object Array
object[] items = new object[2];
//populate our array with the parameters
//passed to our method
items[0] = msg;
items[1] = status;
//call the delegate
_synch.Invoke(_status, items);
}
#endregion

Using this we can update any UI component in the main thread without worrying if the call is being made from the proper thread. So as you can see, there is a thread-safe way to share data between threads. Though we aren’t using data, we’re simply updating a component on the UI with the status of our background thread. This process can now rune while leaving the UI in a usable state.

Basic Instincts: Implementing Callbacks with a Multicast Delegate.

is a follow-up to the December 2002 installment in which I introduced the basic concepts and programming techniques associated with delegates. I am going to assume you have already read that column and that you are familiar with the fundamentals of programming delegates. For example, you should know how to define a delegate type, how to create a delegate object that’s bound to a handler method, and how to execute the handler method by calling the delegate object’s Invoke method.

How to: Combine Delegates (Multicast Delegates)(C#).

This example demonstrates how to compose multicast delegates. A useful property of delegate objects is that they can be assigned to one delegate instance to be multicast using the + operator. A composed delegate calls the two delegates it was composed from. Only delegates of the same type can be composed.

The - operator can be used to remove a component delegate from a composed delegate.

Example

C#

delegate void Del(string s);

class TestClass
{
    static void Hello(string s)
    {
        System.Console.WriteLine("  Hello, {0}!", s);
    }

    static void Goodbye(string s)
    {
        System.Console.WriteLine("  Goodbye, {0}!", s);
    }

    static void Main()
    {
        Del a, b, c, d;

        // Create the delegate object a that references 
        // the method Hello:
        a = Hello;

        // Create the delegate object b that references 
        // the method Goodbye:
        b = Goodbye;

        // The two delegates, a and b, are composed to form c: 
        c = a + b;

        // Remove a from the composed delegate, leaving d, 
        // which calls only the method Goodbye:
        d = c - a;

        System.Console.WriteLine("Invoking delegate a:");
        a("A");
        System.Console.WriteLine("Invoking delegate b:");
        b("B");
        System.Console.WriteLine("Invoking delegate c:");
        c("C");
        System.Console.WriteLine("Invoking delegate d:");
        d("D");
    }
}

Output

Invoking delegate a:
  Hello, A!
Invoking delegate b:
  Goodbye, B!
Invoking delegate c:
  Hello, C!
  Goodbye, C!
Invoking delegate d:
  Goodbye, D!

See Also

Reference

MulticastDelegate

Concepts

C# Programming Guide
Events (C# Programming Guide)

Quick factories using delegates

Delegates are very useful to act as object factories. In the example below, it is used to allow the function to build another object based on it, while keeping the function unaware about types.

namespace Microsoft.MSDN.Wiki.Samples
{
    class Vehicle
    {
        int Price;
    }
    class Car : Vehicle
    {
        protected Car(string model, int passengers)
        {
        }
        public static Car Create(string model, int passengers)
        {
            // Perform some check and create a new instance
            return new Car(model, passengers);
        }
        public Vehicle FakeClone(Vehicle src)
        {
            // Cars are special in our example, and will not be cloned
            // return the original object instead
            return src;
        }
    }
    class Bus : Vehicle
    {
        private string model;
        private int passengers;
        private int cargo;
        protected Bus(string model, int passengers, int cargo)
        {
            // Assign variables
            this.model = model;
            this.passengers = passengers;
            this.cargo = cargo;
        }
        public static Bus Create(string model, int passengers, int cargo)
        {
            // TODO: Perform some check and create a new instance
            return new Bus(model, passengers, cargo);
        }
        public Vehicle CloneSpecial(Vehicle src)
        {
            // TODO: Perform some check and clone the instance
            Bus bus = (Bus)src;
            return new Bus(bus.model, bus.passengers, bus.cargo);
        }
    }
    class Program
    {
        public delegate Vehicle CreateNewVehicle(Vehicle src);
        public void ProcessBus(Bus bus)
        {
            ProcessVehicle(bus, bus.CloneSpecial);
        }
        public void ProcessCar(Car car)
        {
            ProcessVehicle(car, car.FakeClone);
        }
        public void ProcessVehicle(Vehicle source, CreateNewVehicle factory)
        {
            // This function might need to clone the vehicle. Use the delegate
            // so the caller can control the creation
            bool needToCloneObject = true;
            if (needToCloneObject)
            {
                Vehicle newObj = factory(source);
                if (newObj != null)
                {
                    // Now we have a new object to work with
                }
            }
        }
        [STAThread]
        public static void Main(string[] args)
        {
        }
    }
}

As you can see we can take the knowledge about the creation logic outside the function when pure inheritance does not work (e.g. having a generic Clone() function in Vehicle) or when the logic changing depending on the context. For example, you might need to clone in a specific way on one condition and clone it differently on another condition.

Anonymous delegates can be used to simplify context changes

you can also use an anonymous delegate to simplify the cloning process. But remember that this technique is not constrained to cloning. That was just the motivation for the example.

namespace Microsoft.MSDN.Wiki.Samples
{
    class Vehicle
    {
        int Price;
    }
    class Car : Vehicle
    {
        protected Car(string model, int passengers)
        {
        }
        public static Car Create(string model, int passengers)
        {
            // Perform some check and create a new instance
            return new Car(model, passengers);
        }
    }
    class Bus : Vehicle
    {
        private string model;
        private int passengers;
        private int cargo;
        protected Bus(string model, int passengers, int cargo)
        {
            // Assign variables
            this.model = model;
            this.passengers = passengers;
            this.cargo = cargo;
        }
        public static Bus Create(string model, int passengers, int cargo)
        {
            // TODO: Perform some check and create a new instance
            return new Bus(model, passengers, cargo);
        }
        public Bus Clone()
        {
            return new Bus(this.model, this.passengers, this.cargo);
        }
    }
    class Program
    {
        public delegate Vehicle CreateNewVehicle(Vehicle src);
        public void ProcessBus(Bus bus)
        {
            ProcessVehicle(bus, delegate(Vehicle src)
            {
                // Perform some special logic
                return ((Bus)src).Clone();
            });
        }
        public void ProcessCar(Car car)
        {
            ProcessVehicle(car, delegate(Vehicle src)
            {
                // Cars are not cloned in this situation
                return src;
            });
        }
        public void ProcessVehicle(Vehicle source, CreateNewVehicle factory)
        {
            // This function might need to clone the vehicle. Use the delegate
            // so the caller can control the creation
            bool needToCloneObject = true;
            if (needToCloneObject)
            {
                Vehicle newObj = factory(source);
                if (newObj != null)
                {
                    // Now we have a new object to work with
                }
            }
        }
        [STAThread]
        public static void Main(string[] args)
        {
        }
    }
}

SQL Server has a number of new date time formats, but the one I am most interested in is DateTime2. The internal format of the SQL DateTime is commonly mistaken as 2×4 byte integers, with the latter integer being milliseconds since midnight. It is in fact the number of 1/300ths of a second since midnight, which is why the accuracy of the DateTime within SQL Server has historically been 3.33ms. (If you really want to see it, crack it open by converting it to a binary, adding 1 and re-converting, you add 3.33ms, not 1 ms.)

So DateTime2 must use a different format, and as a weekend exercise that had no purpose than understanding the internals I thought I’d take a look. I have not seen the information in the BoL or posted as yet, so might be of use.  I am starting with the DateTime2(7) and looking at the maximum accuracy structure. The code used to crack it open each time is basically as follows:

declare @dt datetime2(7)
set @dt = '2000/01/01 00:00:00'
declare @bin varbinary(max)
set @bin = CONVERT(varbinary(max), @dt)

To make my life easier, SQL conveniently outputs all the values as hexidecimal numbers. The results are not what you would expect.

0x07000000000007240B
0x07000000000008240B

The date which traditionally occupied the first 4 bytes, clearly is occupying the last few bytes. So the format is not going to be obvious or simple. interestingly the returned result is 9 bytes, but the length is quoted as 8. It is returning 8 when checked using the length, that first byte is somewhat odd to make an appearance.  It’s also suspiciously the accuracy value, and with a few tests using a change of accuracy, it show that value changes. So the first pseudo-byte is the accuracy indicator.

To start figuring out some more, let’s take the time back to the beginning point, which in this case is not 1900/01/01 but 0001/01/01 which when converted gives us:

'0001/01/01 00:00:00' => 0x070000000000000000

Start incrementing the day portion and there is an obvious pattern, the 6th byte changes.

'0001/01/02 00:00:00' => 0x070000000000010000
'0001/01/03 00:00:00' => 0x070000000000020000
'0001/01/31 00:00:00' => 0x0700000000001E0000

As you try the 2nd month, to check where the month is, the same byte alters, so it represents days, not specific date parts. Is it the number of days since the beginning of the year? No.

'0001/02/01 00:00:00' => 0x0700000000001F0000

If it was, there would be an issue since 1 byte does not represent enough values, as we can see, FF occurs on the 13th of September, and then it rolls over and puts a 1 in the 7th Byte position.

'0001/09/13 00:00:00' => 0x070000000000FF0000
'0001/09/14 00:00:00' => 0x070000000000000100
'0001/09/15 00:00:00' => 0x070000000000010100

It rolls over, then carries on as before. This immediately suggests the next test, to roll over the year, and the pattern continues.

 '0001/12/31 00:00:00' => 0x0700000000006C0100  
'0001/12/31 00:00:00' => 0x0700000000006D0100

So the format is just counting, we see it in the example as hex, but it is a straight number count going on but the hex values are left-to-right. Only 2 bytes are used so far, which do not represent enough day combinations, add the third byte in by going past 180 years:

'0180/06/06 00:00:00' => 0x070000000000FFFF00  
'0180/06/07 00:00:00' => 0x070000000000000001

So the final byte is then increased, so the number of combinations becomes 16777215 – that seems a lot better and certainly going to cover the range required.

'2001/01/01 00:00:00' => 0x07000000000075250B

So that is the final 3 bytes decoded, a simple pattern - and provides the template of how the time is also stored.

'0001/01/01 00:00:00.0000000' => 0x070000000000000000
'0001/01/01 00:00:00.0000001' => 0x070100000000000000
'0001/01/01 00:00:00.0000255' => 0x07FF00000000000000
'0001/01/01 00:00:00.0065535' => 0x07FFFF000000000000
'0001/01/01 00:00:00.0065536' => 0x070000010000000000

So to check whether the format is the same,

'0001/01/01 00:00:00.9999999' => 0x077F96980000000000

Decode that again and it all matches:

select (152 * 256 * 256) + (150 * 256) + 127
-----------
9999999

When we click over into 1 second exactly, we increment the first byte by 1, so the time portion is still represented in 100ns intervals, with the normal system of each byte counting up 1 every time the previous byte rolls over. As we get to the limit of the 3 bytes, it rolls into the 4th and then the 5th.

'0001/01/01 00:00:01.0000000' => 0x078096980000000000

So the internal format of the DateTime2(7) is decoded, not difficult but it is an interesting choice – it is now a straight binary number, with the Least Significant Byte being on the Left, the Most Significant being on the right (for each section.) Within the byte however, to convert it you must still read it right-to-left.

The first 5 bytes are recording how many time units intervals have passed since midnight, and the last 3 bytes recording how many days have passed since 0001/01/01.

The time unit intervals are dictated by the accuracy of the number, 100ns for DateTime2(7), and 1 Micro second intervals for a DateTime2(6) etc.  The way in which you interpret it does not change, but the units you are multiplying the time portion by, alters based on the accuracy.

You could construct 2 dates that are identical at a binary level, but due to the field meta-data on accuracy, they do not represent the same date time.

 declare @dt1 dt1 datetime2(6)
set @dt1 = '0001/01/01 00:00:00.000001'
declare @dt2 datetime2(7)
set @dt2 = '0001/01/01 00:00:00.0000001'

0x060100000000000000
0x070100000000000000 

 And that is perhaps why on output they automatically have prefixed the binary value with the datetime accuracy, so that they are not entirely identical? I’m not sure but would be interested to find out.

• Its Sunday noon & I’m listening to 15-20 minute odd songs by DT, browsing #interestingness on flickr, randomly tweeting, stumbling.
• My 1st internals got over yesterday.
• I like how all my pending works are done [well, most] & I can sit and do Nothing for the rest of the day.
• Yesterday was one of the most fun filled days of my life in college. We went out to celebrate at Chung Wah and Came back to college and stayed there till evening.
• I like how college is soo much more than school. Its not about “getting up in the morning, going to college, attending classes and coming back home”.
• I got my camera back [which my Dads colleague had borrowed for 2 weeks] yesterday! What timing.
• DT = Adrenaline Rush!!
• My last.fm subscription has still not expired!! It was suppose to have expired months back! #imaybelucky
• I’ve realized making random bullets on blog is much easier than having to talk about one single topic =P

This is more an oddity than anything particularly useful. Sometimes you can’t use the sql*plus DESCRIBE {DESC} command on tables- but you might have an alternative.

I’m doing a lot of work for a client on a 10.2.0.3 database.  I have SELECT ANY DICTIONARY but not SELECT ANY TABLE privilege. This is because there is sensitive data in the database and it is duly protected {and this is certainly not the first time I have worked for a client with full dictionary access but not data access, it’s becoming normal}. I’m granted access to specific things as needs arise.

I knew I had to look at a table called AD_FAKE_STATS_COLUMNS.

select count(*) from mdw.AD_FAKE_STATS_COLUMNS
/
select count(*) from mdw.AD_FAKE_STATS_COLUMNS
                         *
ERROR at line 1:
ORA-00942: table or view does not exist

Have I got the table name wrong?

select table_name,owner
from dba_tables
where table_name = 'AD_FAKE_STATS_COLUMNS'

TABLE_NAME                     OWNER
------------------------------ -----------------
AD_FAKE_STATS_COLUMNS          MDW

OK, it is there, I don’t have access to it. Fine, I don’t want access to the data, I just want to see the structure of the table:

desc mdw.AD_FAKE_STATS_COLUMNS
ERROR:
ORA-04043: object mdw.AD_FAKE_STATS_COLUMNS does not exist

Oh. DESC in sql*plus does not work.

I can’t DESC a table I do not have access to carry out DML on. I’m going to have to go and ask someone to give me permission to see the table. How annoying.

Or do I?

@tab_desc
Enter value for tab_name: AD_FAKE_STATS_COLUMNS
old 21: where table_name like upper (nvl('&TAB_NAME','W')||'%')
new 21: where table_name like upper (nvl('AD_FAKE_STATS_COLUMNS','W')||'%')

TAB_OWN  TAB_NAME              COL_NAME              M COL_DEF
-----------------------------------------------------------------
MDW      AD_FAKE_STATS_COLUMNS TABLE_NAME            Y VARCHAR2(30)
                               COLUMN_NAME           Y VARCHAR2(30)
                               COPY_STATS_FROM       N VARCHAR2(61)
                               LOW_VALUE_SQL         N VARCHAR2(100)
                               HIGH_VALUE_SQL        N VARCHAR2(100)
                               DISTINCT_SQL          N VARCHAR2(100)
                               DAYS_HIST_NULL_AVGLEN N NUMBER(3,0)

I have access to the data dictionary. So I can see the structure of the table, which after all is what I wanted and is what the client is happy for me to have.{I’ve never much liked the DESC command in sql*plus, I replaced it with a little sql script against the data dictionary years ago}.

In case you want it, here is the script:

-- tab_desc.sql
-- Martin Widlake date? way back in the mists of time
-- my own replacement for desc.
-- 16/11/01 improved the data_type section
 SET PAUSE ON
 SET PAUSE 'Any Key...>'
 SET PAGES 24
col Tab_own form A10
col tab_name form a22 wrap
col col_name form a28 wrap
col col_def form A14
 break on tab_own skip 1 on tab_name skip 1
 spool tab_desc.lst
 select
 owner                               Tab_Own
,table_name             Tab_Name
,column_name            Col_Name
,decode(NULLABLE,'Y','N','Y') Mand
,data_type||decode(data_type
       ,'NUMBER','('
        ||decode(to_char(data_precision)
                ,null,'38'
                ,     to_char(data_precision)||
                      decode(data_scale,null,''
                                      ,      ','||data_scale)
                 )
                    ||')'
       ,'DATE',null
       ,'LONG',null
       ,'LONG RAW',null
,'('||Substr(DATA_LENGTH,1,5)||')'
				     )  col_def
from dba_tab_columns
where table_name like upper (nvl('&TAB_NAME','WHOOPS')||'%')
order by 1,2,column_id,3,4
/
spool off
clear col
--

Hi,

Here the placement for Deputy Manager-International Sales & Marketing.

International Sales & Marketing ( P Band Sales and Mktg)

KRA’s:

• Throughput Contribution.
• Capital Efficiency.
• Operating Expense Control.

Job Description:

The vacancy is in the Sales & Marketing department of PED.
The incumbent will be responsible for:

Essential Qualification:

• B.E / B.Tech/ Diploma in (Mech /Prod/ Chem / Metallurgy)

Essential Experience :

Preferred Experience:

• Critical Process Equipment.

Special Skills Required:

Request you to furnish following details in the CV:
1) Year of passing and percentage in SSC.
2) Year of passing and percentage in HSC / Diploma
3) Year of passing and percentage in B.E.
4) Current Location.
5) Current and Expected CTC.
6) Contact Details (Mobile number, landline number and e-mail id)

Would you be interested ?

Please send us your latest updated profile with contact nos.current & expected salary details and joining time required to priya@kbsconsultants.com

You may also suggest this opening to your friends who may be interested.

Further Information:

KBS Consultants
Flat H,Kulothungan Apts,
No, 5 Natesan Road
Ashoknagar,
Chennai 600 083.India
Phone: +91-44 2489 5341 / 2371 9622

Visit Our Sites:

International jobs: http://www.jobsearchworld.com/

SAP ERP Jobs : http://www.jobsvista.com/

Core Engineering Jobs : http://www.gotachance.com/

Technology Jobs : http://www.kbsconsultants.com/

India Jobs : http://www.kbsconsultants.net.in/

http://www.kbsconsultants.org.in/

I’ve got a couple more postings on Data Dictionary performance to get where I plan to go to with this, but if you want to deep-dive into far more technical details then go and check out Dion Cho’s excellent posting on fixed object indexes.

I was not planning on getting into the sys.x$ fixed objects as you need SYS access to look at them, which not everyone has, but this is where Dion goes. His posts are always good, I need to check them far more.

As a soft-technical aside, I often mention to people when doing courses on SQL or writing standards or even the odd occasions I’ve discussed perception, that we Westerners are taught to read from left to right, top-to-bottom and we pick out left justification very well. Code laid out like the below we find easy to read:

select pers.name1                surname
      ,pers.name2                first_forename
      ,pers.nameother            middle_names
      ,peap.appdate              appointment_date
      ,decode (addr.name_num_ind
                 ,'N', to_char(addr.housenum)
                 ,'V', addr.housename
                 ,'B', to_char(addr.housenum
                              ||' '||addr.housename)
                                 house_no_name
      ,addr.address2             addr_street
      ,addr.address3             addr_town
      ,addr.address4             addr_dist
      ,addr.code                 addr_code
from person              pers
     ,address            addr
    ,person_appointments peap
where pers.addr_id     =addr.addr_uid
and   pers.pers_id     =peap.pers_id
and   pers.active_fl   ='Y'
and   pers.prim_cons   ='ANDREWSDP'
and   peap.latest_fl   ='Y'

But this is not true of other cultures, where people do not read left to right, top to bottom. I have had this confirmed just a couple of times when people who were born in Eastern cultures are in the course/conversation.

So I was very interested to see Dion’s Korean version of the blogpost I reference above (I really hope this link here to the korean version is stable).
The main body of the page is on the right, not left, but the text appears to be left justified.

Of course, I am horribly ignorant, I do not know which direction Koreans read in . I could be spouting utter rubbish.

A couple of posts ago I demonstrated that the DBA_ views are general faster than the ALL_ views and that adding an owner filter to DBA_ views is potentially quicker than selecting from the USER_ views.

Something else I do automatically, to help performance, is always specify and link via the schema owner.

The below is based on a 10.2.0.3 database with fixed object statistics gathered, though a long while ago. A have a test partitioned table called TEST_P and I want to see how many blocks are in the table and also in the table partitions, so I link between DBA_TABLES and DBA_TAB_PARTITIONS. I only specify the table name as I know there is only one table called TEST_P:

select dbta.table_name,dbta.blocks
       ,dtp.partition_name,dtp.blocks
from dba_tables         dbta
    ,dba_tab_partitions dtp
where dbta.table_name=dtp.table_name
and   dbta.table_name='TEST_P'

TABLE_NAME     BLOCKS PARTITION_     BLOCKS
---------- ---------- ---------- ----------
TEST_P           7394 ID_MAX            370
TEST_P           7394 ID_40K           1756
TEST_P           7394 ID_30K           1756
TEST_P           7394 ID_20K           1756
TEST_P           7394 ID_10K           1756
5 rows selected.
Elapsed: 00:00:00.04

Statistics
---------------------------------------------
          0  recursive calls
          0  db block gets
        782 consistent gets

Now I do the same but specify the schema owner in the join.

select dbta.table_name,dbta.blocks
      ,dtp.partition_name,dtp.blocks
from dba_tables         dbta
    ,dba_tab_partitions dtp
where dbta.table_name =dtp.table_name
AND   DBTA.OWNER      =DTP.TABLE_OWNER
and   dbta.table_name ='TEST_P'

TABLE_NAME     BLOCKS PARTITION_     BLOCKS
---------- ---------- ---------- ----------
TEST_P           7394 ID_MAX            370
TEST_P           7394 ID_40K           1756
TEST_P           7394 ID_30K           1756
TEST_P           7394 ID_20K           1756
TEST_P           7394 ID_10K           1756
5 rows selected.
Elapsed: 00:00:00.04

Statistics
--------------------------------------------
          0  recursive calls
          0  db block gets
        580 consistent gets

Just for the record, I have run these statements several time so the consistent gets are constant. The form with the “AND DBTA.OWNER =DTP.TABLE_OWNER” line is consistently running at 580 consistent gets and the version without at 782 consistent gets.

If I now add in the where clause to look only for tables within the schema I am interested in, the code is even more efficient:

select dbta.table_name,dbta.blocks
      ,dtp.partition_name,dtp.blocks
from dba_tables         dbta
    ,dba_tab_partitions dtp
where dbta.table_name =dtp.table_name
AND  DBTA.OWNER       =DTP.TABLE_OWNER
and   dbta.table_name ='TEST_P'
and   dbta.owner      =user

TABLE_NAME     BLOCKS PARTITION_     BLOCKS
---------- ---------- ---------- ----------
TEST_P           7394 ID_10K           1756
TEST_P           7394 ID_20K           1756
TEST_P           7394 ID_30K           1756
TEST_P           7394 ID_40K           1756
TEST_P           7394 ID_MAX            370
5 rows selected.
Elapsed: 00:00:00.03

Statistics
--------------------------------------------
          0  recursive calls
          0  db block gets
        137  consistent gets

Down to 137 consistent gets.

It’s maybe not at all suprising that fully specifying the schema owner of the table we are interested in allows a more efficient path through the data dictionary, but it is something we all often fail to specify as it means writing more SQL, and it works quickly enough. For one table.
But then if you convert a quick check script into part of your system housekeeping scripts and then use that housekeeping script on a very large database, things can start to run very slowly. I’ve never seen that happen, not with one of my own scripts, honest

The reason to always specify the schema owner, at all times when possible, when drilling down the data dictionary, is that so many things link back to sys.obj$ and it has an index with a leading column on owner {well,owner#}. The whole data dictionary tends towards 3rd normal form and parent-child relationships through IDs (name#, obj#, user#, ts#). Schema and name identify the objects and are one of the few links into this structure, which then links to other tables via obj# and other hash IDs.

IND_NAME           TAB_NAME           PSN       COL_NAME
--------------- ------------------ --------- -------------
I_OBJ1          OBJ$               1         OBJ#

I_OBJ2          OBJ$               1         OWNER#
                                   2         NAME
                                   3         NAMESPACE
                                   4         REMOTEOWNER
                                   5         LINKNAME
                                   6         SUBNAME

I_OBJ3           OBJ$              1       OID$

So my advice is, if you are looking at data dictionary views where the schema owner is involved, always include that column in the join and always filter by schema owner if you can. And if things are still slow, pull out the view code and look at the indexes on the underlying sys.$ views. After all, the data dictionary is, to all intents and purposes, just a normal normalised database.

This blog is not about blog comments. It’s about table and column comments in the data dictionary.

Some of you may well be going “huh?”. Others are probably going “Oh yes, I remember them?”. Table and column comments appear to be suffering the same fate as ERDs, who’s slow demise I bemoaned a couple of weeks ago. They are becoming a feature not known about or used by those with “less personal experience of history” {ie younger}.

It’s a simple principle, you can add a comment against a table or a column,up to 4000 characters. {you can also add comments against index types, materialized views and operators (huh?), at least in 10.2}.

comment on table widlakem.person is
'Test table of fake people for training purposes, approx 50k records'
Comment created.

select * from dba_tab_comments
where owner='WIDLAKEM'
and table_name = 'PERSON'

OWNER      TABLE_NAME                     TABLE_TYPE
---------- ------------------------------ -----------
COMMENTS
--------------------------------------------------------------------------
WIDLAKEM PERSON TABLE
Test table of fake people for training purposes, approx 50k records

--

comment on column widlakem.person.second_forename is
'null allowed, second or middle name. If more than one, delimited by / character'

select * from dba_col_comments
where owner='WIDLAKEM' and table_name = 'PERSON'
and column_name='SECOND_FORENAME'

OWNER      TABLE_NAME                     COLUMN_NAME
---------- ------------------------------ ------------------------------
COMMENTS
-------------------------------------------------------------------------------
WIDLAKEM PERSON SECOND_FORENAME
null allowed, second or middle name. If more than one, delimited by / character

So you can stick basic information into the data dictionary where it will remain with the object until the object is dropped or the comment is updated. (You can’t drop a comment, you can only update it to ”:

>comment on table widlakem.person is ”;

It’s simple, it’s sensible, it’s solid.

And it seems to be dying out. In fact, I had stopped adding comments to my tables and columns as no one else seemed to bother. Probably as a consequence of them not being added, no one ever seemed to think to look at them to get hints about the database structure and table/column use.

But Raj sitting next to me is as old a hand at this game as myself and I “caught” him adding comments to the little schema we are working on together. Well, if he is bothering, so will I!

How about Oracle Corp? How do they manage on this front? After all, the Oracle Reference manual has all these short descriptions of tables and columns in the data dictionary {some helpful, some utterly unhelpful}:

select owner,count(*) from dba_tab_comments
where owner in ('SYS','SYSTEM','SYSMAN')
group by owner
OWNER        COUNT(*)
---------- ----------
SYSTEM            151
SYSMAN 472
SYS 3894

3 rows selected.

Heyyyy, nice Oracle.

select owner,table_name,comments
from dba_tab_comments
where owner in ('SYS','SYSTEM','SYSMAN')

OWNER      TABLE_NAME
---------- ------------------------------
COMMENTS
-----------------------------------------------------
SYS ICOL$

SYS CON$

SYS FILE$

SYS        UET$

SYS        IND$
SYSTEM     MVIEW_RECOMMENDATIONS
This view gives DBA access to summary recommendations
SYSTEM     MVIEW_EXCEPTIONS
This view gives DBA access to dimension validation results
SYSTEM     AQ$_QUEUE_TABLES
SYS        SEG$

SYS        COL$

SYS        CLU$
SYSTEM     SQLPLUS_PRODUCT_PROFILE

SYSTEM     PRODUCT_PRIVS

SYSTEM     HELP

SYSMAN MGMT_NOTIFY_QTABLE

SYSMAN AQ$MGMT_NOTIFY_QTABLE_S

Oh. Lots of blanks. Not so nice Oracle. No, scrub that, several lines are not blank, so Not a bad attempt Oracle.

Why all the blanks? Why have Oracle set blank comments? That’s because a blank table comment gets created when you create a table, and a blank column comment is created per column.

create table mdw_temp (col1 number);
Table created.

select * from dba_tab_comments where table_name = 'MDW_TEMP';
OWNER      TABLE_NAME                     TABLE_TYPE
---------- ------------------------------ -----------
COMMENTS
----------------------------------------------------------------------
WIDLAKEM   MDW_TEMP                       TABLE

1 row selected.

select * from dba_col_comments where table_name='MDW_TEMP';
OWNER      TABLE_NAME                     COLUMN_NAME
---------- ------------------------------ ---------------
COMMENTS
-------------------------------------------------------------------------
WIDLAKEM   MDW_TEMP                       COL1

1 row selected.

So what populated system-like comments do we have?

select owner,count(*) from dba_tab_comments
where owner in (‘SYS’,'SYSTEM’,'SYSMAN’)
and comments is not null
group by owner

OWNER COUNT(*)
———- ———-
SYSTEM 73
SYSMAN 15
SYS 944

OK, there are some, and as you can see below, some are more useful than others…

OWNER      TABLE_NAME
---------- ---------------
COMMENTS
----------------------------------------------------------------------------------------------------
SYS        SYSTEM_PRIVILEG
           E_MAP
Description table for privilege type codes.  Maps privilege  type numbers to type names
SYS        TABLE_PRIVILEGE
           _MAP
Description table for privilege (auditing option) type codes.  Maps privilege (auditing option) type
numbers to type names
SYS        STMT_AUDIT_OPTI
           ON_MAP
Description table for auditing option type codes.  Maps auditing option type numbers to type names
SYS        RESOURCE_MAP
Description table for resources.  Maps resource name to number
SYS        SESSION_PRIVS
Privileges which the user currently has set
SYS        SESSION_ROLES
Roles which the user currently has enabled.
SYS        ROLE_SYS_PRIVS
System privileges granted to roles
SYS        ROLE_TAB_PRIVS
Table privileges granted to roles
SYS        ROLE_ROLE_PRIVS
Roles which are granted to roles
Oracle_DatabaseInstance contains one entry for each Oracle Instance that is
centrally managed.  A Real Application Cluster has one entry for each of the
instances that manipulate it.  Instances of Oracle_DatabaseInstance are created
using the database instances that are known to the Oracle Enterprise Manager
repository.
SYS        DBA_AUDIT_OBJECT
Audit trail records for statements concerning objects, specifically: table, cluster, view, index, se
uence, [public] database link, [public] synonym, procedure, trigger, rollback segment, tablespace, r
ole, user
SYS        USER_AUDIT_OBJECT
Audit trail records for statements concerning objects, specifically: table, cluster, view, index, se
uence, [public] database link, [public] synonym, procedure, trigger, rollback segment, tablespace, r
ole, user
SYSMAN     ORACLE_DATABASESTATISTICS
Oracle_DatabaseStatistics provides current information about the statistics for
a database.  Database statistics pertain to the database and have the same
value regardless of the database instance that is used.
SYSMAN     ORACLE_DBINSTANCESTATISTICS
Oracle_DBInstanceStatistics contains statistics for a database instance.  These
are retrieved from the Oracle Managment Repository that is managing the
database upon request from a managment client.

If you don’t add comments to tables and comments, you will just have blank entries for them in the data dictionary.

So why not pop a few real comments in there, especially for any tables or comments where the name is not really telling you what that column or table is for? It’s free and easy, and it might just prove useful. And if you add them to your tables, I’ll add them to mine.

From version 10 onwards, Oracle introduced the concept of the recycle bin and the purge command. I won’t go into details of the recycle bin, go to Natalka Roshak’s FAQ page on it if you want the details.

I am only concerned with the fact that from Oracle 10 onwards by default if you drop a table or partition it is not dropped but just renamed, unless you use the PURGE command {which can be tricky if you are using a 9i client as PURGE won’t be accepted by SQL*Plus}.

And it keeps catching me out. So I’m blogging about it –  more for me than any of you lot ).

How exactly does it catch me out? Well, as an example,  I’m looking at some partitioning considerations at the moment and ran the below.

select count(distinct(table_name)) from dba_tables
where partitioned='YES'
and owner='ERIC'

COUNT(DISTINCT(TABLE_NAME))
---------------------------
393
Elapsed: 00:00:00.14

Statistics
----------------------------------------------------------
8 recursive calls
0 db block gets
7079 consistent gets

select count(distinct(table_name))
from dba_part_tables
where owner='ERIC'

COUNT(DISTINCT(TABLE_NAME))
---------------------------
419
Elapsed: 00:00:00.07

Statistics
--------------------------------------------
8 recursive calls
0 db block gets
14084 consistent gets

419 records from DBA_PART TABLES and 393 from DBA_TABLES.

How can that be? How can you have more records with partition information than tables with PARTITIONED flag set to YES?

{And for readers of my post yesterday on querying the data dictionary, check which of the statements performs better – one might expect the view specific to partitions to be quicker than one that is not, but then the partition-specific view is looking at a lot more under the covers}.

OK, Let’s find the objects in DBA_PART_TABLES not in DBA_TABLES

SELECT DISTINCT TABLE_NAME FROM DBA_PART_TABLES
WHERE OWNER ='ERIC'
minus
SELECT DISTINCT TABLE_NAME FROM DBA_TABLEs
WHERE PARTITIONED='YES'
AND OWNER='ERIC'

TABLE_NAME
------------------------------
BIN$c0wxEyj93/vgQKAKQstN+w==$0
BIN$c59mEBlzOvLgQKAKQssjNA==$0
BIN$c6D0zY7yTrvgQKAKQssmdQ==$0
BIN$c6FJQBU6FG3gQKAKQssxCw==$0
BIN$c9qFcYojb/LgQKAKQssyVQ==$0
BIN$c9qFcYokb/LgQKAKQssyVQ==$0
BIN$c9qFcYolb/LgQKAKQssyVQ==$0
BIN$c9qFcYoqb/LgQKAKQssyVQ==$0
BIN$c9t8m70OdEXgQKAKQss4LA==$0
BIN$c9tu0S7KoUngQKAKQss39w==$0
BIN$c9uGpjGSbcvgQKAKQss4fw==$0
BIN$cWfis1j/BGPgQKAKQss7Lw==$0
BIN$cWft6d2mNcvgQKAKQss7qQ==$0
(snip)
BIN$czz0jembWe/gQKAKQsthhg==$0
BIN$czzu+/hC+yTgQKAKQsthpw==$0

26 rows selected.

All 26 records are for deleted objects. You can tell they are deleted as the name is “BIN$something”.

That’s the thing that keeps catching me out, I keep forgetting to exclude deleted but not purged objects from maintenence scripts. *sigh*.

DBA_TABLES has a DROPPED column but other views do not, so you have to exclude on the name, which is not ideal. Someone might want to create a table called “BIN$somthing”.

Mind you, the DROPPED column is not much help:
select count(*) from dba_tables WHERE DROPPED !=’NO’

COUNT(*)
———-
0

So that is as much use a chocolate teapot.
{some little bell is ringing in my head that you need special privs to see the dropped tables. I have DBA role and I can’t…}

This does highlight something very, very odd though.

Why have the records gone from DBA_TABLES and not DBA_PART_TABLES?

Because the concept of the wastebasket has not been around long enough with V10.2 of Oracle for it to work consistently

I’ll check DBA_OBJECTS for the last object name in the above list:

select * from dba_objects
where object_name='BIN$cxIzaly77FzgQKAKQst5tg==$0'

OWNER OBJECT_NAME SUBOBJECT_NAME OBJECT_ID
--------------------------------------------------------
DATA_OBJECT_ID OBJECT_TYPE CREATED LAST_DDL_TIME
--------------------------------------------------------
TIMESTAMP STATUS T G S
---------------------------
ERIC BIN$cxIzaly77FzgQKAK 2613938
Qst5tg==$0
TABLE 08-SEP-2009 13:49 08-SEP-2009 14:10
2009-09-08:14:10:02 VALID N N N

And I can still look at it {note the use of quotes to allow for an object name with mixed case}:

DESC ERIC.”BIN$cxIzaly77FzgQKAKQst5tg==$0″
Name Null? Type
—————————————– ——– ————
LOAD_LOG_ID NUMBER(12)
LOG_DATE DATE
ABCDEFG_ID NUMBER(12)
REMOTE_ADDRESS VARCHAR2(20)
KGBHFDERS NUMBER(12)
PRODUCT_ID NUMBER(12)
KNGFDRET NUMBER(22)
WKNHGFYTRERDS NUMBER(12)
CHANNEL_TYPE VARCHAR2(1)
COUNT_HITS NUMBER(12)
MFDKEMNFK NUMBER(12)
COUNT_NON_WEIGHTED NUMBER(12)

I can still select from it with the right privileges as well {I’ve not shown this, I’m out of time}.

In conclusion:

• Objects that are dropped but not purged are still in the data dictionary.
• You may have to exclude such objects using the fact that the name starts ‘BIN$…’.
• Dropped objects appear in some parts of the data dictionary but not others.
• You can still and in fact look at dropped objects.
• I have a poor memory.

Which data dictionary view you use to look at information about your schemas and objects can be important from a performance point of view.

{Part two is here.}

Try this:

set autotrace on statistics
set timi on
set pause off

select count(tablespace_name) from user_tables;
select count(tablespace_name) from all_tables;
select count(tablespace_name) from dba_tables;

You might want to execute each select statement a couple of times until the consistent gets are stable.

These are the stats off my test system, version 10.2.0.3:

USER_TABLES
8 records
0.14 seconds
14217 consistent gets

ALL_TABLES
4455 records
0.34 seconds
29097 consistent gets

DBA_TABLES
4455 records
0.21 seconds
16834 consistent gets

The select against user_tables is fastest and uses the least consistent gets as the view implicitly limits the search to your objects with the clause “where o.owner# = userenv(‘SCHEMAID’)”. You use the USER_ views when you really only want to see your objects.

More interesting are the timings for ALL_TABLES and DBA_TABLES. I have SELECT_ANY_TABLE and SELECT_ANY_DICTIONARY so I see all tables with either view. But the DBA_ view is quicker and takes less consistent gets than the ALL_ view.

The reason is that the ALL_ views take into account checking your privileges to see if you have the right to see any particular object. The DBA_ views do not as you can only see them if you have system privileges to do so.

ALL_TABLES includes the following to check for access rights:

and (o.owner# = userenv('SCHEMAID')
or o.obj# in
(select oa.obj#
from sys.objauth$ oa
where grantee# in ( select kzsrorol
from x$kzsro
)
)
or /* user has system privileges */
exists (select null from v$enabledprivs
where priv_number in (-45 /* LOCK ANY TABLE */,
-47 /* SELECT ANY TABLE */,
-48 /* INSERT ANY TABLE */,
-49 /* UPDATE ANY TABLE */,
-50 /* DELETE ANY TABLE */)
)
)

So, if you are using an account that has access rights to see everything (DBA access typically), use the DBA_ views and they will perform better.

If you only want your objects then the best performance will come from using the USER_ views and not the ALL_ views, correct? It makes sense! Remember my cautions in earlier posts about theory and practice. There is a sting in this particular tale…

select count(tablespace_name) from user_tables;
COUNT(TABLESPACE_NAME)
———————-
8
Elapsed: 00:00:00.15

Statistics
——————————————————
0 recursive calls
0 db block gets
14217 consistent gets

Compared to

select count(tablespace_name) from all_tables
where owner=USER;
COUNT(TABLESPACE_NAME)
———————-
8
Elapsed: 00:00:00.10

Statistics
—————————————————–
0 recursive calls
0 db block gets
4117 consistent gets

selecting from ALL_TABLES and filtering by USER performs significantly better on my system than using the USER_TABLES view!

One caveat. I suspect my test system has poor fixed-object statistics, so I would be interested if anyone else finds different results on their system.