Hello Friends.

One of the most common question that one can ask during formatting hard disk drive is about the partition format. “Which partition format to use out of NTFS and FAT32 , isn’t it?”

For many of you this question remains unsolved.So, let’s try to solve this problem right now. Truel speaking  there’s no hard and fast rule to format our hard disk drive. But files systems NTFS and FAT 32 have certain advantages. Example NTFS is more reliable in terms of security and provide more flexibility to the system , Whereas in  some system configurations FAT32 is
a master choice.Also if you are using windows 98 its advisable to go with FAT32 System because Windows 95/98/Me is unable to deal with NTFS Format.

Now let’s talk about Security. Using NTFS volume,its easier to restrict access to files and folders by using permissions option but on a FAT or FAT32 drive, anyone with physical access to your computer can access any files stored on that drive that i am sure you will never want if the document is secret or important.

DISK Error……. ….. ……

You might have seen Disk error check during the start of your system. NTFS volumes have ability to recover from disk errors more readily as compare to your FAT/FAT32 drive. Hence Windows XP repair file system errors automatically when the system is restarted (in case of NTFS).  On the other hand  FAT and FAT32 drives are more prone to disk related errors.

Partition Size o you know  the maximum partition size for a FAT32 drive created by Windows XP can only be upto 32 GB whereas a single NTFS volume can carry 16,384 GB (16
terabytes) and also there is a good scope if you want to expand storage on existing drive letters. Interestingly you need not to take  back up, repartition, reformat or  restore in doing  so.

Best of luck

Partitions are often used to help increase the performance of SQL select activity via the concept of partition exclusion – the CBO will be able to identify which of the partitions could hold the data of interest and ignore the others. This is their main benefit from a SQL Select performance perspective.

The problem is, the partitioning key is not always included in your SQL statements.

I discussed this potential negative impact of partitioning on SQL select performance in this post and this one also. I did not give any proof then, so the first half of this post provides that.

As an example, I have created a table with 45 partitions, partitioned on the column ID, an ascending numeric primary key (traditionally sourced from a sequence). There is an index on the ID column, locally partitioned. The table also has a column CRE_DATETIME, which is the date the record was created. This column is also indexed with a locally partitioned index.

NAME                           VALUE
------------------------------ -------------
compatible                     10.2.0.3.0
cpu_count                      8
db_block_size                  8192
db_file_multiblock_read_count  16
optimizer_mode                 ALL_ROWS
sga_target                     0
sort_area_size                 65536

create table test_p3
(id           number(10) not null
,cre_datetime date not null
,status        number(1) not null
,num_1         number(4) not null -- random 20
,num_2         number(4) -- random 500
,num_3         number(5) -- cycle smoothly
,num_4         number(5) -- Random 10000
,vc_1          varchar2(10)
,vc_2          varchar2(10)
,vc_pad        varchar2(2000))
tablespace users
partition by range (id)
(partition id_01k values less than (1000)
tablespace users
,partition id_02k values less than (2000)
tablespace users
,partition id_03k values less than (3000)
tablespace users
...
,partition id_45k values less than (45000)
tablespace users
--
,partition id_max values less than (maxvalue)
tablespace users
)
/
select table_name,partitioning_type p_type
,partition_count pt_count
from dba_part_tables
where owner=user and table_name = 'TEST_P3'
TABLE_NAME                     P_TYPE    PT_COUNT
------------------------------ ------- ----------
TEST_P3                        RANGE           46

>@ind_cols
IND_NAME           TAB_NAME           PSN       COL_NAME
------------------ ------------------ --------- ------------
TP3_PK             TEST_P3            1         ID
TP_CRE_DT          TEST_P3            1         CRE_DATETIME

{This is a very typical senario, as is the opposite situation where the table is partitioned on date but has a numeric column holding eg order number or customer ID.}

The below shows a select of a single record by ID.

select id,cre_datetime,num_1,vc_1
from test_p3 where id=37123
/
        ID CRE_DATETIME           NUM_1 VC_1
---------- ----------------- ---------- ----------
     37123 12-OCT-2009 02:52         19 RMZAN
1 row selected.

Execution Plan
----------------------------------------------------------
----------------------------------------------
| Id  | Operation                          | Name    | Rows  |
Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
----------------------------------------------------------------
----------------------------------------------
|   0 | SELECT STATEMENT                   |         |     1 |
  22 |     2   (0)| 00:00:01 |       |       |
|   1 |  PARTITION RANGE SINGLE            |         |     1 |
  22 |     2   (0)| 00:00:01 |    38 |    38 |
|   2 |   TABLE ACCESS BY LOCAL INDEX ROWID| TEST_P3 |     1 |
  22 |     2   (0)| 00:00:01 |    38 |    38 |
|*  3 |    INDEX UNIQUE SCAN               | TP3_PK  |     1 |
     |     1   (0)| 00:00:01 |    38 |    38 |
----------------------------------------------------------------
----------------------------------------------
Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
          3  consistent gets
          0  physical reads

Note the PARTITION RANGE SINGLE and pstart/pstop are both 38. Consistent gets are only 3. The query is very efficient as the CBO identified that records matching the WHERE clause could only exist in the one partition. The relevant local index is examined for 2 consisted gets (the BLEVEL of the local index partitions is 1) and then one consistent get against the table.

Now I’ll scan for records for a range of IDs:

select id,cre_datetime,num_1,vc_1
from test_p3 where id between 30980 and 31019
/
        ID CRE_DATETIME           NUM_1 VC_1
---------- ----------------- ---------- ----------
     30980 25-SEP-2009 01:20         15 JQLYA
     30981 25-SEP-2009 01:24          5 WYWCR
     30982 25-SEP-2009 01:28         10 QZSHD
     30983 25-SEP-2009 01:32         18 KQBSU
     30984 25-SEP-2009 01:36          2 HRPMA
....
     31018 25-SEP-2009 03:52          3 YOPJO
     31019 25-SEP-2009 03:56         10 GNGKG

40 rows selected.

Execution Plan
----------------------------------------------------------
----------------------------------------------
| Id  | Operation                          | Name    | Rows  |
Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
----------------------------------------------------------------
----------------------------------------------
|   0 | SELECT STATEMENT                   |         |    40 |
 880 |     9   (0)| 00:00:01 |       |       |
|   1 |  PARTITION RANGE ITERATOR          |         |    40 |
 880 |     9   (0)| 00:00:01 |    31 |    32 |
|   2 |   TABLE ACCESS BY LOCAL INDEX ROWID| TEST_P3 |    40 |
 880 |     9   (0)| 00:00:01 |    31 |    32 |
|*  3 |    INDEX RANGE SCAN                | TP3_PK  |     2 |
     |     2   (0)| 00:00:01 |    31 |    32 |
----------------------------------------------------------------

Statistics
---------------------------------------------
          0  recursive calls
          0  db block gets
         52  consistent gets
          0  physical reads

Again, you can see partition exclusion, this time as a PARTITION RANGE ITERATOR and Pstart/Pstop of 31 and 32. Consistent gets were higher, but I did fetch several records.

Now a query is issued for a record with a specific value for CRE_DATETIME.

select id,cre_datetime,num_1,vc_1
from test_p3 where cre_datetime =
   to_date('06-OCT-2009 12:20','DD-MON-YYYY HH24:MI')
/
        ID CRE_DATETIME           NUM_1 VC_1
---------- ----------------- ---------- ----------
     35105 06-OCT-2009 12:20         11 JPQHO
1 row selected.

Execution Plan
----------------------------------------------------------
------------------------------------------------
| Id  | Operation                          | Name      | Rows  |
 Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
----------------------------------------------------------------
------------------------------------------------
|   0 | SELECT STATEMENT                   |           |     1 |
    22 |    48   (0)| 00:00:01 |       |       |
|   1 |  PARTITION RANGE ALL               |           |     1 |
    22 |    48   (0)| 00:00:01 |     1 |    46 |
|   2 |   TABLE ACCESS BY LOCAL INDEX ROWID| TEST_P3   |     1 |
    22 |    48   (0)| 00:00:01 |     1 |    46 |
|*  3 |    INDEX RANGE SCAN                | TP_CRE_DT |     1 |
       |    47   (0)| 00:00:01 |     1 |    46 |
----------------------------------------------------------------

Statistics
---------------------------------------
          1  recursive calls
          0  db block gets
         90  consistent gets
          0  physical reads

Notice that partition exclusion does not occur, you see PARTITION RANGE ALL, Pstart/Pstop of 1 – 46. The CBO cannot identify which partitions may or may not hold records with that CRE_DATETIME. Consistent gets are now a lot higher, 90 (as compared to 3 for the situation where partition exclusion can occur}, as oracle has to prob each and every local index partition to identify if any records for the given value exist.

Of course, you probably would not look for a record with a specific datetime, but in the opposite senario of having partitioned on datetime, it would not be at all unusual to look for a record with a given ID and partition exclusion not being possible.

As I mentioned earlier, this need to scan all of the local indexes because no partition exclusion is possible is something I discussed a couple of weeks back.

Now I will select for the range of dates that my range scan in ID identified:

select id,cre_datetime,num_1,vc_1
from test_p3
where cre_datetime between to_date('25-SEP-2009 01:20','DD-MON-YYYY HH24:MI')
                   and     to_date('25-SEP-2009 03:56','DD-MON-YYYY HH24:MI')

        ID CRE_DATETIME           NUM_1 VC_1
---------- ----------------- ---------- ----------
     30980 25-SEP-2009 01:20          3 BGAFO
     30981 25-SEP-2009 01:24         15 PXGJD
     30982 25-SEP-2009 01:28         12 PGQHJ
     30983 25-SEP-2009 01:32         17 TIBZG
     30984 25-SEP-2009 01:36         11 EQQQV
...
     31018 25-SEP-2009 03:52         18 FSNVI
     31019 25-SEP-2009 03:56         16 LJWNO
40 rows selected.

Execution Plan
----------------------------------------------------------
------------------------------------------------
| Id  | Operation                          | Name      | Rows  |
 Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
----------------------------------------------------------------
------------------------------------------------
|   0 | SELECT STATEMENT                   |           |    42 |
   924 |    54   (0)| 00:00:01 |       |       |
|   1 |  PARTITION RANGE ALL               |           |    42 |
   924 |    54   (0)| 00:00:01 |     1 |    46 |
|   2 |   TABLE ACCESS BY LOCAL INDEX ROWID| TEST_P3   |    42 |
   924 |    54   (0)| 00:00:01 |     1 |    46 |
|*  3 |    INDEX RANGE SCAN                | TP_CRE_DT |    42 |
       |    47   (0)| 00:00:01 |     1 |    46 |
----------------------------------------------------------------
------------------------------------------------

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
        136  consistent gets
          0  physical reads

That actually selects back the same records, but now a PARTITION RANGE ALL is invoked, Pstart/Pstop os 1-46 and the number of consistent gets goes up to 136.

I discussed the impact of partitions on range scans in this second post about performance issues with partitions.

Thankfully, you can often alleviate this problem of considering all partitions.

The below shows a scan for records for a range of dates.

select count(*)
from test_p3
where cre_datetime between to_date('18-OCT-2009','DD-MON-YYYY')
                   and     to_date('20-OCT-2009','DD-MON-YYYY')

  COUNT(*)
----------
       721
1 row selected.

Execution Plan
----------------------------------------------------------
----------------------------------
| Id  | Operation            | Name      | Rows  | Bytes | Cost
(%CPU)| Time     | Pstart| Pstop |
----------------------------------------------------------------
----------------------------------
|   0 | SELECT STATEMENT     |           |     1 |     8 |    49
   (0)| 00:00:01 |       |       |
|   1 |  SORT AGGREGATE      |           |     1 |     8 |
      |          |       |       |
|   2 |   PARTITION RANGE ALL|           |   731 |  5848 |    49
   (0)| 00:00:01 |     1 |    46 |
|*  3 |    INDEX RANGE SCAN  | TP_CRE_DT |   731 |  5848 |    49
   (0)| 00:00:01 |     1 |    46 |
----------------------------------------------------------------
----------------------------------

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
         90  consistent gets
          0  physical reads

As you can see, no partitiion exclusion is possible, every partition is considered (PARTITION RANGE ALL, Pstart/Pstop of 1-46). 90 consistent gets are required.

The trick is to introduce a second WHERE clause, limiting the query by the partitioning key as well.
As a human, you can recognise that the ID and the CRE_DATETIME columns are going to increase pretty much in synchronisation. Records with a CRE_DATETIME a month ago are going to be found in partitions with a lower ID range than those from a week ago. Oracle does not know this business logic, so you have to tell it.

Let us say you never have more than 1,000 records created a day. so if you want “today’s” data, you need only consider IDs that are between the max(ID) and the max(ID) less 1,000. To be safe, you would increase this range substantially. The idea is to exclude most partitions without risking missing data:

select count(*)
from test_p3
where cre_datetime between to_date('18-OCT-2009','DD-MON-YYYY')
                   and     to_date('20-OCT-2009','DD-MON-YYYY')
and  id between 39000 and 41000

  COUNT(*)
----------
       721
1 row selected.

Execution Plan
----------------------------------------------------------
------------------------------------------------
| Id  | Operation                   | Name             | Rows  |
 Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
----------------------------------------------------------------
------------------------------------------------
|   0 | SELECT STATEMENT            |                  |     1 |
    13 |    14   (8)| 00:00:01 |       |       |
|   1 |  SORT AGGREGATE             |                  |     1 |
    13 |            |          |       |       |
|*  2 |   VIEW                      | index$_join$_001 |    34 |
   442 |    14   (8)| 00:00:01 |       |       |
|*  3 |    HASH JOIN                |                  |       |
       |            |          |       |       |
|   4 |     PARTITION RANGE ITERATOR|                  |    34 |
   442 |     7  (15)| 00:00:01 |    40 |    42 |
|*  5 |      INDEX RANGE SCAN       | TP_CRE_DT        |    34 |
   442 |     7  (15)| 00:00:01 |    40 |    42 |
|   6 |     PARTITION RANGE ITERATOR|                  |    34 |
   442 |     9  (23)| 00:00:01 |    40 |    42 |
|*  7 |      INDEX RANGE SCAN       | TP3_PK           |    34 |
   442 |     9  (23)| 00:00:01 |    40 |    42 |
----------------------------------------------------------------
------------------------------------------------

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
         18  consistent gets
          0  physical reads

In the above I have managed to exclude most of the partitions and still get the correct data, by adding the clause and id between 39000 and 41000 . There is something a little unusal {if you have not seen it before} with the plan being scans of two indexes and then hashed together and viewed, but both are PARTITION RANGE ITERATOR scans and Pstart/Pstop values are 40-42. Consistent gets are down to 18.

You do have to take great care with this method that the extra WHERE clause you add on the partitioning key will never exclude records you want to find. But so long as you do, it is a very powerful technique.

I’ll go into that a little more in the next posting {which hopefully will not be as delayed as this one!}

In a couple of recent posts on partitions, on single row and range scan access I have started exploring the performance of partitions. I will continue on the main theme soon, honest.

I feel a little bad that I have not given concrete examples so far {creating very large objects for the sake of demonstrations and proof-of-concepts is a bit of an issue with VLDBs, due to the space and time requirements to do it}. So, here is just some real-world information on index depth, or BLEVEL of indexes on Very Large DataBases.

The BLEVEL is the number of levels in the index. Minus one. See this post for a full description, but in brief, if an index has a BLEVEL of 3, that means there is a root node, a level of branch nodes, a second level of branch nodes and then the leaf nodes. 4 levels. 4 IOs to check if the index holds the value you are looking for. And then one or more IOs against the table to get the actual records from the table, if required (one block if there is only one row, one or more blocks if there are several rows).

The below is from a 25TB datawarehouse system. I am looking for the maximum index depth on the database:-

select max(blevel) from dba_indexes

MAX(BLEVEL)
———–
4

OK, how many indexes are this deep?…

select owner,index_name from dba_indexes
where blevel=4

OWNER INDEX_NAME
—————————— ———————-
ERIC W_IIIIIIII_URLS_PK

1 row selected.

Oh. I have a pretty massive database and the deepest index I have is BLEVEL 4 and there is one of them. So the index height is 5. And access to that table for a given referral ID is going to take 5 IOs to work down the index and one against the table.

What is more, the index is an IOT

INDEX_NAME INDEX_TYPE
—————————— —————————
W_REFERRAL_URLS_PK IOT – TOP

so, in fact, to get to the table record it is just the 5 IOs as the table record is held in the IOT.

So with a 25TB data warehouse, the deepest normal indexes are BLEVEL 3.

select index_name,num_rows/leaf_blocks
from dba_indexes
where BLEVEL =3
order by num_rows desc

INDEX_NAME                             NUM_ROWS  LEAF_BLOCKS
------------------------------ ---------------- ------------
W_WL_SH_API_XIIIIIIIIIII_ID     4,585,108,192   13,406,015
W_WL_SHIIIIIIIIIIIXXXX_ID    4,564,350,002   16,892,898
W_WL_SIIIIIIIIIIIIIIIIIIIIII_ID     4,422,775,820   17,189,536
W_WL_SHIIIIIIIIIIIIIIIIIIIIII_ID    4,182,087,545   12,243,438
W_WL_GX_RESIIIIIIIIIII_LOG_ID1     3,690,374,216   14,613,388
IDX_W_WL_LIIIIIIIIIIIESS_ID          351,507,905    1,958,109
WL_LOGIN_SUIIIIIIIIIIIIE_ID          348,004,861    1,550,180
IND_IIII_LEG_ID                     312,727,000      972,690
IND_TMLR_MARIIIIIIIIIII_ID           306,988,247    1,121,592
PK_TBL_IIIIIIIIIIIUNNER              305,566,240      939,775
PK_IIIIIIIIIIIGS                    284,208,000      863,000

I asked an old friend, Tony Webb, to check out the depth of indexes on the largest databases they have, which range from a few TB to several 10s of TB. {one  is 100TB plus, but a lot of the data is in LOBs, so the actual number of records is only billions, not 10’s of billions } No index had a BLEVEL greater than 3.

Hi Martin,

I ran the first query on DAVE, DEE(our biggest db), DOZY and MICK. The max depth results were 3,2,3 and 3. Our largest database only had 2. In fact some of the other largish dbs also only have a max depth of 2.

That enough for your blog Martin?

Now, some of you may be smelling a Rat, and you would be right. At both sites, the VLDBs have been designed with partitioning as a key part of the Physical Implementation. All the largest objects are partitioned. What impact does partitioning have on BLEVEL?

The depth of the index segment is dependent on the number of keys per block.

Let us say you have 8k blocks {this is a questionably low block size for a data warehouse, but it is what some of these systems have} and your index is on a couple of columns, totalling 20 bytes per key on average, including rowid. Allowing block overhead, that is approx 8000/20 entries = 400 max….

Blow that, that’s whooly theory, I’ll look at my real 25TB database for number of rows per leaf block… 

select leaf_blocks, num_rows\leaf_blocks rows_per_leaf
from dba_ind_statistics
where num_rows > 100000000
order by leaf_blocks desc

 LEAF_BLOCKS ROWS_PER_LEAF
------------ -------------
  30,211,949    325.602273
  28,717,447    262.762331
  26,308,608    382.505959
  25,429,862    385.047045
  24,054,626     309.92864
  23,660,747    382.571434
  22,458,058    411.063205
  22,316,976    346.620625
  18,875,827    379.952198
  17,451,901    338.600909
  17,189,536     257.29466

From the above let us take a low figure of 300 average enteries per block. For a BLEVEL 0 , a one block index, that would be 300 rows that are referenced. For a level 1 index, 300 entries in the root block will point to 300 leaf blocks, so that will be 300*300 rows…90,000 entires.

Go down another BLEVEL to 2 (and a height of 3) and you have a root node referencing 300 Branch nodes, referenceing 300 Leaf nodes each referencing 300 records. 300*300*300 = 27 million.

Another level (BLEVEL 3), another factor of 300 and that is 8.1 billion.

BLEVEL 0= 300
BLEVEL 1 = 90,000
BLEVEL 2 = 27,000,000
BLEVEL 3 = 8,100,000,000
BLEVEL 4 = 6,480,000,000,000 (6,480 billion entries).

You can see that with an 8k block size and being very pessimistic about the number of entries per block (I used a low average to calculate enttries per block where as in reality most indexes will fill the block at the root and branch nodes to close to full before throwing a new level) your have to increase your data volume by amost 300 to throw a new level of index.

On average, knowing nothing about the actual number of records in an index, you would have to reduce your number of indexed rows by 150 times (half of 300) to have a 50% chance of lowering the BLEVEL of a locally partitioned index by one. ie 150 or more partitions.

with a 32k block size, you would have to reduce your data volume by more like 620 times to drop the BLEVEL (not 600 as you have less wastage from block overhead with larger blocks).

To reduce it by two it would be 300*150 I think, wiht 8k block size {can any statisticians, like Mr Lewis, help me out on this one?}. If I am right, that is 45,000 partitions. In that 25TB datawarehouse, no table has more than 4,000 partitions.

That is why I say, when you partition a table and have locally partitioned indexes, you might reduce the index level by 1. Might.

However, none of these systems has more than 4000 partitions per table. That might sound a lot, but it is only going to reduce an index BLEVEL by 1. Almost certainly it will, but if you have read the previous postings, that is not going to really help index lookups be that more efficient

I’ll just add a couple of last comments.

• If you have an index BLEVEL of 4 or 5 or more and do not have a VLDB, you might want to look at why {hints, it could be an IOT, it could be an index on several concatenated VARCHAR2 columns, it could be you are using 2k block size, it could be that you regularly delete a large pecentage of your data and then re-insert more data, it could be you have a one-table database, hehe.}.
• Point one does not mean you should rebuild said index. I almost never rebuild indexes. The need to do so is one of those Oracle myths. There are situations where index rebuilds are advantageous, but they are pretty damned rare. If you regularly rebuild all your indexes or you are thinking of doing so, can I suggest you book a “meeting” for an afternoon and sit down with a cup of tea and google/bing/yahoo or whatever and check out index rebuilding first? Ignore any sites you land on offering database health checks or training courses on cruise liners.

That’s all for now.

Reste plus qu’à travailler

Cliquez ici pour télécharger le concertino (pdf).

An interesting thing happened to me today. I added a field to a fact table that I had. As normal, I did the change in my data source view as procedure dictates, and create the new measure on my cube. Everything looked fine and dandy until I tried to deploy the cube. Right off the bat, this error came up: “OLE DB error: OLE DB or ODBC error: Invalid column name ‘<ColumnName>’.; 42S22″. Huh? What happened? I double checked the DSV and the cube (thou not entirely as we’ll see further down) and everything looked ok. Re-deploy, re-discover same error.

A Google browse and an hour later, I read on www.sqlservercentral.com forums that it could be the partitions. And then the proverbial lightbulb turned on in my head. I forgot to recheck the partitions, to include the field in the select statements for the fact table. Lesson learned? If you use partitions, always check them when you add a field to your fact table.

Presque aussi bien que les 1er versions de Sibélius, mais ici totalement gratuit

MuseScore est un éditeur WYSIWYG de partitions multiplateforme sous licence GNU GPL. Quelques précisions:

partition de guitare réalisée avec Muse Score

Vous pouvez la télécharger sur le lien ci-contre

A télécharger ici

la bionda treçça

la bionda treçça

partition de guitare réalisée avec Muse Score
Vous pouvez la télécharger sur le lien ci-contre

If you have installed Windows 7 or Server 2008 R2 through either the Windows media or MDT then you have noticed that a 100 – 300 MB system partition is created that is hidden when you finish in the final operating system (well it is not really hidden, it just doesn’t have a drive letter).  The Microsoft Windows team made that decision so that bitlocker could easily be enabled at a later time without having to repartition the disks.  However if you don’t ever plan on enabling Bitlocker on your Windows 7 or Server 2008 R2 systems you can use MDT with an extra variable in the customsettings.ini and MDT won’t create the extra partition.  The variable is:

DoNotCreateExtraPartition = YES

This is a follow up to my first posting on why Partitions are not for Performance where I discuss how lookups against a partitioned table with a locally partitioned index and no partition pruning can lead to performance problems. Basically, the CBO ends up scanning all local indexes to find the record(s) you want, which with a hundred partitions will probably result in several hundred buffer gets instead of 4 or 5.

The first posting dealt with the simple situation where a single record is being selected but this lack of partition pruning also crops up with range scans of course.

{warning, this is a long post, but if you use partitions you need to understand this}.
{Caveat, it has been pointed out the last comment, and by association this one, appears partition-negative. I am not partition-negative, partitions are great but for reasons generally other than performance and, to improve performance, you need to understand how partitions fit in with the CBO}

Imagine you have an unpartitioned ORDERS table with 10 million rows, the primary key is the traditional ascending numeric, derived from a sequence.
About 5,000 orders a day are received, the table goes back around 10 years (as in the past, less orders per day were received).
There is also an ORDER_DATETIME column, a DATE type column holding the data and time the order was placed. This column has a traditional index on it, which has a B-level of 3.

You want to query for all ORDERS placed on a date, say 1st November 2009, so you issue a query with the following WHERE clause:
WHERE ORDER_DATETIME >= TO_DATE(‘01-NOV-2009′,’DD-MON-YYYY’)
AND ORDER_DATETIME < TO_DATE(‘01-NOV-2009′,’DD-MON-YYYY’)

That will select around 5,000 out of 10 million rows and so a nested loop lookup on the index is the most suitable {just accept this for now, OK?}

The CBO will work down the levels on the index on ORDER_DATETIME to the first record of interest, taking 4 I/Os. It will then scan the index for all entries in that range, so maybe 10 leaf blocks, and then read the database blocks for each one – 5000 I/Os against the table, but as the data will be clustered well, most reads will be to the same say 100 blocks, so they will be cached in the block buffer cache. {Under Oracle – to 11.1 anyway – all IO is treated as physical, so the default optimizer will calculate the cost based on this, but in reality most IO will be logical IO, not physical IO}.

So the real cost will be 4 IOs to find the start of the index to scan, 10 I/Os to scan the index and 5000 IOs to get the table data – 5014 logical I/O’s, with between 0 and (4+10+100 =114) physical IOs to get the data into the cache, depending on what was cached already.

OK, remember those figures…

Now we partition the table by the ORDER_DATETIME column into 100 partitions and the index on ORDER_DATETIME is locally partitioned of course. Your same WHERE clause on ORDER_DATETIME will now resolve to a single index partition.

If you are lucky the local index partition will have a B-level of 2, 1 less than the standard non-partitioned index. So, with the same WHERE clause, the CBO will work down the local index partition to the first record of interest, for 3 I/Os. Then it will scan the index for the same 10 leaf blocks and find 5,000 records to check in the table partition, with the same 5,000 I/Os. You have saved….Yep, 1 logical IO. Physical IOs are between 0 and (3+10+100 =113). Let’s be kind and assume you saved a physical IO.

In this case, partitioning saved 1 out of 5104 logical and 1 out of 114 physical IOs.

Partitioning, in this case, is performance agnostic. It matters very, very little.

In reality, if your application is working on the same day of data over and over again, using the one partition, then that set of data will be cached and a nested-loop access to it will find the data in memory -but that would be true with the standard table and index too So still no gain….

I’ve assume a nested-loop access path is still used by the CBO when accessing your partition. This is where things can be very different. {and it is also where I should swap from theory and start creating some test data to prove this, but I’ll hang myself out first and then look like an idiot next week when I am back with my test system }.

Let’s estimate that the unpartitioned table is 100,000 blocks (100 rows per block).
With 5,104 I/Os against a 10 million row table/index , a nested loop lookup is likely to be chosen as the CBO estimated the number of I/Os to be less than scanning the whole table (100,000 blocks divided by the actual multi block read count {which is not the db_file_multi_block_read_count, but I’m skipping over that detail}, let’s say 12, for say 8,500 I/Os ).

With the table split into 100 partitions, there is 1,000 blocks in each partition. The whole table can be scanned with 1,000/real-multi-block-read-count {12}. So 80 or so I/Os

So in reality, your partitioning MAY be beneficial as the CBO decided to forget the index and nested lookups and simply scans the whole partition into memory for maybe 80-100 I/Os using multi block scans and processes the data in memory, burning some CPU as well, to find the records you want.

So, instead of 5,103 IOs the CBO decides to do 80-100 I/Os and some extra memory/cpu work, which is cheap given the power of modern CPUs.

So, partitions may help performance as the CBO swaps from nested loop lookups with an index to simply full scanning the partition.

You may start to wonder if that index on ORDER_DATETIME helps much… If it is to support range scans of a significant number of records, maybe not, but if it helps single or small-number record lookups, the index still helps.

I don’t know about you, but my brain is full for the night. I’ll leave this for another day….

There is a myth that Partitions in Oracle magically aid performance, even a general assumption that the main role of partitioning is to aid performance. This is not so. There are certainly many cases where Partitioning can aid performance but in my experience they just as often hinder rather than help overall performance.

The myth is so strong that when asked at interview, most DBAs {and even some performance specialists} will site performance as the main (and occasionally only) benefit of partitioning.

Why can partitions hinder performance? Let’s say for example that you have a 10 million row table and an index on that table. That index has a B-Level of 3, which means it has a root block, one level of branch nodes, a second layer of branch nodes and then the leaf-node level. To access a row in the table via the index Oracle needs to read the root block, two branch blocks and then a leaf block to get the rowid of the record. This allows the table block {and from that the row} to be read. This is depicted in the below diagram, the numbered orange squares represent the blocks as selected in turn from the index and then table:

That is 5 I/O operations to access that row.

Now we partition the table into 5 partitions, 2 million rows each. The index is locally partitioned. If you are lucky, you may, just may, end up with local index partitions with a B-level 1 less then the original table, so in our case a B-level of 2. The often suggest process is now that one partition is considered and thet the CBO will read one root node, a branch level block, a leaf block and then the block from the partition.

4 I/Os and a saving of 20% {I’m ignoring for now caching and whether it is physical or logical IO}.

A saving of 20% IF {and only if} you have local index partitions with a lower B-Level than the equivalent non-partitioned index. And the access is to one partition alone.

However, I keep seeing situations where the index look up does not include the partition key. So you get the below situation:

The CBO cannot exclude any partitions so it has to scan each one. For most partitions, maybe for all but one, no records are found, but the index has to be checked with 3 IO operations each. so in my example 16 I/Os are done to get the one record of interest.

16 I/O operations instead of 5.

The situation is often not spotted, at least initially, as the time taken to carry out the extra local index partition scans is “small”, especially for specific lookups. Usually any testing is done on a table with only a small numer of partitions.

I remember well the first time I came across this {on an Oracle 9.0 database I think}, there was well over 100 partitions and a process that checked many thousands of individual records had slowed down significantly, taking 4 or 5 times as long as before.

An indicator that the problem is occurring is when a single record lookup against the index and then table is taking perhaps several dozen to several hundred consistent gets rather than the 5 or 6 it should. Also, you will see the partition itterator in the explain plan. In that first case I came across the issue, consistent gets of about 380 per record fetched were being seen for a select that returned 1 record 99% of the time from a single table lookup. I’ve since seen the same problem on tables with over a thousand partitions, each local index being scanned for a total of almost 5000 consistent gets per record.

You may think that this would be an unusual situation as access against very large tables is either full/partial table scans or lookups on the PK/with a WHERE clause including the partitioning key – but it is actually very common. Partitioned tables are being used more and more in large but generally OLTP applications or sets of records are identified in a datawarehouse that are then checked more specifically with generally row-specific logic. With VLDBs which have many, many partitioned tables with many, many partitions each, the problem is common and often not recognized in the fog of other issues and massive I/O levels.

I’ve only covered a general performance issue with partitions here, I’ll expand on the theme and this simplistic example in the next post.

And yes, there are many ways partitioning CAN aid performance. I aim to get to those too. I really love partitioning.

There are a huge variety of storage systems  available on the market it is vital that businesses make the most of the options available to them, select the most appropriate products for their needs and maximise their space.

These can be; storage bins which can handle large and small capacities, warehouse racking that will add a second or third level to a workspace, stillages which can provide storage for difficult objects or even partitions to make the most of existing space.

All of these plus many more storage options can help you and your business.

That all architectural structures are covered by solid opaque walls on all four sides to maintain privacy and security is a given. So, a home with walls of glass that offer a clear view of the inside is bound to create more than just a ripple, and 128 Werner Sobek does just that. This rather novel concept in residential architecture seems to have broken through all conventions and set a new – if not higher – standard in innovative construction.

R128 Werner Sobek is four-storey house, floating high on a hill overlooking the city of Stuttgart, Germany. Inside this curious creation, there are no doors, switches, interior walls or partitions and no closed rooms. But what takes the cake is that the home generates its own energy. The construction material used to create R128 is one hundred percent eco-friendly and recyclable.

R128 has a most attractive glass façade and is devoid of a basement. High quality triple-glazed glass with inert gas filling is in use. The use of a modular design, complete with glass panels and steel frames ensured easy assembly and disassembly of the construction. The insulated glass panels prevent overheating of the interior during summer and loss of energy and warmth during winter. The supporting steel frame comprises of 10 tons of steel.

The ceiling of the Werner Sobek glass house consists of prefabricated panels overlaid by plastic. Beneath the unscrewed floor, aluminium ceiling panels are affixed by clip connections. Lighting, heating and cooling systems are fitted into that layer and this acts as an acoustic absorber pattern.   

Sensor controlled doors have been installed on the upper and lower levels of the house. All appliances and environmental systems are also controlled by motion sensors and voice commands. The front door has a voice recognition feature which allows it to open automatically on a voice command. Water faucets in the bathrooms are regulated by sensors. Windows are controlled by touch screen technology.

Every floor has two folding windows each, which allow natural daylight and fresh air to enter the house. During summer, cool water running through the floor elements removes excess heat from the entire house with the help of a heat exchanger. Thus surplus energy is stored for use in winter. This ensures minimal energy consumption. 48 solar powered modules with a total capacity of 6.2 KW are installed on the rooftop, which are responsible for supplying all the power required by the pump system.

This green show-house is expected to go a long way in promoting energy-efficient architecture.

Today we spent a great deal of time investigating partitions and equivalence relations. In particular we came to the conclusion that partitions and equivalence relations on a set are really the same thing. That is, for every partition, we have an equivalence relation, and for every equivalence relation, we get a partition.

We also continued our exploration of the number systems as a partition of the integers. These numbers, as well as many other types of equivalence classes and partitions will be very important later in the semester. We investigated these early to start thinking about them as soon as possible.

We did not introduce any further homework problems due this Friday. However, I did mention that you can try Section 0: 18 as a bonus problem.

I’ve recently had to format my Ubuntu partition because of a Kernel Panic. This happened when I hit the “suspend” button in Boxee. The OS freezed still (it didn’t suspend) and there was no way to resurrect it than resetting the system. When I rebooted the there came the Kernel Panic. During the brief unsuccessful efforts to fix the problem I checked the fstab file to see if Boxee did somehow mess with it. I did not know much about fstab so, now that my system is fresh again, I want to discuss the wonders of fstab, this tiny yet superimportant text file.

fstab stands for file system table and it’s basically a configuration file which gives the kernel all the necessary information to mount partitions (usually only the internal ones; external partitions like usb flash drives and similar are mounted via volume managers). Being a configuration file fstab is located under the /etc directory; to read it you can just locate and open it by the graphical interface, however only root can write to it so if you need to modify it you must access with su (super user) privileges.

Disclaimer: Critical system files in Linux can only be modified by ROOT and not by the user(s). The rationale behind this is to make the system more safe against security attacks and mishandling by users. Having said that, use super user privileges to access system files only if you fall into one of the following categories:

- You know exactly what you’re doing

- You’re in an experiment environment

- Your system is completely and –by your point of view- irrevocably compromised so you judge you have nothing to lose

To read-only your fstab gile in console type:

~$ cat /etc/fstab

To open the fstab file in write mode open a console and type:

~$ sudo gedit /etc/fstab

As I said before the information contained in fstab is plain text, organized in rows and columns and laid out as follows

[device] [mount point] [file system type] [options] [dump] [pass]

[device] > indicates the device or partition that contains the file system
Note for Ubuntu: Canonical’s distro now uses UUID (Universally Unique Identifier) by default to identify partitions. In order to listyour devices by UUID you can use ~$ sudo blkid

[file system types] > indicates the type of file system. On Ubuntu you will likely see ext3 or ext4 for the main partition where root resides. Depending on the type of partitions or devices in your computer you could encounter any of the following file system types:

• ext2, ext3, ext4, reiserfs (don’t miss the macabre gossip related to this one http://en.wikipedia.org/wiki/Hans_Reiser), etc.
• auto (commonly used for removable devices in order to try to detect the specific file system of the device)
• vfat (identify FAT partitions)
• ntfs (identify NTFS partitions)
• udf,iso9660 (identify optical media -aka CD/DVD-)
• swap (a hard disk partition where a page of memory is copied to free up that page of memory)
• procfs (stands for process file system and it’s a pseudo file system (dynamically generated at boot) used to access process information from the kernel. The file system is often mounted at the /proc directory and occupies no storage space

[options] > there are several options which specify how the partition is handled and what can or cannot do. I’ll try to cover them briefly here and let you google for them if you need an in depth description.

• auto / noauto specify if the partition has to be automatically mounted or not at boot or when mount –a command is executed. auto itself it’s almost useless because it’s the default behavior. noauto is more interesting as it blocks a partition to be mounted unless you manually mount it.
• sync / async are for I/O to the file system and indicate whether it has to be done synchronously (at the same time a I/O operation is issued) or asyinchronously. The meaning of this gets more clear if you imagine to issue a command to execute data transfer to a floppy disk or rw optical media.
• exec / noexec is meant either to allow or prevent execution of binaries within the partition. If you set you’re the partition where root dwells as noexec your system is screwed
• dev / nodev interpret or not interpret character or block special devices on the file system
• ro / rw is used to specify if the partition is read-only or read-write mounted
• suid / nosuid allow or block operations of suid or sgid bits. Suid and sgid stand for set user id and set group id. When a suid or sgid file is executed, the process which runs it is granted access to system resources based on the user who owns the file and not the user who created the process.
• user / nouser are used to specify the user related permissions to mount the file system. user means that all the users can mount the file system and by consequence of this the noexec, nosuid,nodev options are automatically implied. nouser means instead that only root can mount the file system and this is the default setting.
• defaults uses the default settings which are rw, suid, dev, exec, auto, nouser, async.
• _netdev is for a network device which has to be mounted after bringing up the network and it’s only valid for nfs (Network File System)

Note for Ubuntu: from the 8.04 release Ubuntu uses relatime as default for linux native file systems. This options replaces atime and it’s used to know when a file was last used. Read more about atime and relatime here.

[dump] > this field specifies if the backup utility dump is enabled -1- or not -0-. Dump is a application which backs up and restores file systems as a whole. This field is usually set to 0.

[pass] > this field is related to the fsck system utility. fsck, which stands for “file system check”, is a program which check the state of the file system. In case of a system crash or a forced shutdown fsck

should detect it and then proceed automatically or user guided to fix the damaged parts. The options are the following:

• 0 -> don’t check
• 1 -> check this partition first
• 2 -> check this partition next

fsck checks the file system every 30 mount operations as default. It’s possible to modify this interval (mostly by using tune2fs)

Here belows my fstab file commented by the system. Try to read it and retrieve the various specifications and options described before. You will surely encounter elements not covered in my description of fstab – i.e. ‘sw’, ‘utf8’ ‘errors=remount-ro’, ecc..- as you possibly will by reading your own fstab. This is common because technologies are for their nature prone to rapid changes and, like systems, tends to accumulates details in times. We have caught a couple of them like relatime and UUID while for the few remaining others you can google them to find more information or issue the command ~$ man fstab in a terminal

# /etc/fstab: static file system information.
#
# Use ‘vol_id –uuid’ to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# / was on /dev/sda5 during installation
UUID=197ac49c-16f9-41a6-9c18-40e87231f4ba / ext3 relatime,errors=remount-ro 0 1
# swap was on /dev/sda6 during installation
UUID=84365e80-cbc7-495f-9e46-46a852bef3c2 none swap sw 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0

My wife has taken on the challenge of homeschooling our children this year. My main participation in this is a weekly math session with the kids in the evening on any subject of my choosing!

Tonight was our first session. I decided to do partitions with them. I am priming them to be able to help me with my research on Q1 graphs.

We pulled out cubical blocks and I told the kids to make partitions with them. The hardest part about this is keeping my mouth shut and staying out of their way.

B invented Ferrers diagrams. Meanwhile I set A to work on making all partitions of the small numbers. She found 1 partition of 1, 2 partitions of 2 and 3 partitions of 3. She started working on 4 and found 4 partitions. Then B chimed in with a 5th partition of 4. This upset A and she refused to accept it as a partition, because it didn’t follow the pattern that she had seen. She stormed off, but came back and was ready to accept the 5th partition of 4.

I tried to get them thinking about how we could know that we had got all of them. They haven’t come up with anything along those lines yet. B (of his own volition) started working on an algorithm to generate all of the partitions of a given number. The algorithm needs work, so far only generating the n partitions of n: (1,1,…,1), (2,1,1,..,1), (3,1,1,..,1),…,(n-1,1), (n). He was also working on an algorithm for getting partitions of n+1 from partitions of n. I think that one was pretty incomplete too.

Their minds were still going, but it was getting late, so I sent them to bed. It was a good evening of math.

My eyes go right to the no-no spot. Can't help it.

So, a month in. Over sixty entries between the two of us.

10 things I learned.

1.I kvetch a lot about giving up my passions; it’s just barking. Never going to happen.

2.Writing about writing is almost as satisfying as actually writing.

3.Almost.

4.The minute you show contempt for someone else’s passion, you undercut your own.

5.You have to be comfortable with the things you love. Treating comics like something that belong hidden under your mattress just leads to odd, misplaced guilt.

6.Discomfort also drives passion; we’re only motivated to be something new when we’re unsatisfied.

7.I can’t stop writing. No matter what. It’s a problem. But it’s also a pile o’ good.

8.The people you love have to be a part of your passion. You can’t pick and choose amongst a series of partitions.

9.If you indulge solely for someone else’s sake, it’s really hard to find a foothold for your passion. But not impossible.

10.I like making people laugh. I like the venting this blog affords. And I like it best when the two meet.

Tomorrow, we start phase two: Get It Together, a month of organization, lifehacking, new systems, and bad habits.

See you then, for 28 Until 30…

GlassBuild America is very proudly echoing its name as a very well known trade show in the United States. GlassBuild America placed itself in the list unveiled by the Tradeshow Week magazine, as one of the top 200 trade shows in North America. GlassBuild America is an affair, which largely gathers the glass, doors and windows professionals in North America. This grand affair is going to be held from 30^th September to 2^nd October – 2009 at Georgia World Congress Center, Atlanta, GA.

Organizers

This event is organized by the National Glass Association along with its partner organizations namely the American Architectural Manufactures Association, the Glass association of North America, the Insulating Glass Manufactures Alliance and the Bath enclosure Manufactures Association, GlassBuild America. This event unites the large number of audience not found anywhere else. GlassBuild America will definitely help the exhibitors business to move to the next higher level.

Thousands of industry professionals

More than 9000 professionals from various industries are assembling in GlassBuild America, every year. The same case is expected in this year too. Decision makers from all facets of the glass, windows and doors industries, from several companies with a great target of achieving commercial and architectural markets to manufacturers focusing on products for the construction and remodeling of the residents attend this GlassBuild America.

Exhibit space and Exhibits

GlassBuild America nearly covers an exhibit area of about 200,000 square-feet. This trade show features the exhibits covering the wide spectrum of glass, window and door industries. The Exhibits includes glass processing equipment, windows and door manufacturing equipments, partitions and also the latest technologies for all types of glass and fenestration products.

Industry leaders

GlassBuild America is the one and only shows conducting in the combination with the Industry’s leading publications – Glass Magazine and Window & Door and with the industry’s major trade associations – National Glass Association (NGA), American Architectural Manufactures Association (AAMA), Glass Association of North America (GANA), Insulating Glass Manufacturers Alliance (IGMA) and Bath Enclosure Manufacturers Association (BEMA)

Exhibitor profile

Exhibitors will exhibit their various products are from different departments. The exhibitor profile includes curtain walls and storefront system suppliers, glass manufacturers and fabricators, suppliers of all types of flat glass processing equipments, suppliers of machinery of fabricating aluminum, vinyl and other types of framing materials, windows and doors system suppliers, insulating glass machinery manufacturers, bath enclosures and mirror manufacturers, solar panel manufacturers and suppliers and so on.

Visitor profile

Visitors profile includes manufacturers/fabricators of commercial and/or glass products, manufacturers of residential glass windows and glass doors, glazing contractors, dealers and distributors of residential windows and doors, dealers or retailers of commercial and/or residential glass products, distributors/wholesalers of commercial and/or residential glass products, architects or contractors and so on.

Global Fair Alliance

GlassBuild America has combined with Fensterbau/Frontale, fenestration china and Istanbul window to form the Global Fair Alliance on April 24^th, 2008 at Washington. The main motive of this alliance is to help the exhibitors and visitors to easily recognize those four organizations as the world’s leading events for windows and façade industries.

Paragon is a ntfs reader/writer for the mac and usulaly costs $60 but you can get it free today! …well i got it yesterday. i deno if it still works today but its worth a shot!

but yeah, you should totally get it. especially if you want to be able to read/write to a windows partition/harddrive

http://www.v3.co.uk/articles/download/2207317