Showing posts with label AWR Report. Show all posts
Showing posts with label AWR Report. Show all posts

Friday, December 6, 2013

OS Statistics from AWR Reports

There are occassions when it is important to review the "Operating System Statistics" section of the AWR Report. Examples can be :
1. The Database under review is not the only database on the server and one or more other database(s) and/or other application(s) are contending for CPU (and I/O !) resources. If you are reviewing an AWR report for a set of jobs which are CPU intensive (eg high "consistent gets", high "latch waits" and "latch spins" etc), then it is also important to know if adequate CPU was available for the database. You may or may not have been informed of the other database(s) and/or application(s) running on the same server before you begin reviewing AWR reports.
2. The application tier for the database under consideration is on the same server. When you are investigating high "SQL*Net message from client" waits for batch jobs where you know that neither network latency nor user think time would be the cause of these waits, you need to know if CPU resources are being consumed by the application as well.

Pre-10g, there was no method to obtain OS performance statistics from within Oracle. StatsPack would show elapsed time and Oracle time but not any information about CPU usage outside of 'CPU used by this session' and 'CPU used when call started' -- ie CPU usage reported against the same database's sessions.

10g AWR Reports do provide Operating System statistics.

Here are some examples.

Example 1 :
For these 11.8 minutes :

              Snap Id      Snap Time      Sessions Curs/Sess
            --------- ------------------- -------- ---------
Begin Snap:       211 24-Feb-08 22:45:15        21       3.1
  End Snap:       212 24-Feb-08 22:57:03        18       1.9
   Elapsed:               11.80 (mins)
   DB Time:                0.31 (mins)

we see :


Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
RMAN backup & recovery I/O            1,681          82     49  437.9 System I/O
control file sequential read          3,800          20      5  109.4 System I/O
control file parallel write             526           5      9   24.2 System I/O
CPU time                                              3          16.2
db file sequential read                 236           2      8    9.6   User I/O

Statistic Name                                       Time (s) % of DB Time
------------------------------------------ ------------------ ------------
sql execute elapsed time                                 15.5         82.7
inbound PL/SQL rpc elapsed time                           3.5         18.6
PL/SQL execution elapsed time                             3.3         17.5
DB CPU                                                    3.0         16.2
RMAN cpu time (backup/restore)                            2.0         10.8
parse time elapsed                                        0.5          2.9
hard parse elapsed time                                   0.5          2.6

So, Oracle CPU time was only about 3 seconds. Was the server also doing something else ?
These are the Operating System statistics from the AWR Report :


Statistic                                       Total
-------------------------------- --------------------
AVG_BUSY_TIME                                  43,445
AVG_IDLE_TIME                                  27,531
AVG_SYS_TIME                                    4,977
AVG_USER_TIME                                  38,468
BUSY_TIME                                      43,445
IDLE_TIME                                      27,531
SYS_TIME                                        4,977
USER_TIME                                      38,468
...
NUM_CPUS                                            1

These OS CPU statistics account for 709.76seconds(BUSY+IDLE time).
If Oracle accounted for 3seconds of CPU time, then it seems as if some other processes accounted for 381.68 seconds(384.68-3).

Also, the Operating System statistics show that the CPU utilisation [including %sys] was 61.21% (434.45 of 709.76 seconds).

Example 2:
For these 3 hours :


              Snap Id      Snap Time      Sessions Curs/Sess
            --------- ------------------- -------- ---------
Begin Snap:       213 25-Feb-08 00:01:00        18       1.9
  End Snap:       216 25-Feb-08 03:00:40        18       1.9
   Elapsed:              179.67 (mins)
   DB Time:                0.09 (mins)

we see :


Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
control file sequential read          6,096           7      1  124.0 System I/O
control file parallel write           3,593           4      1   77.7 System I/O
CPU time                                              3          50.4
db file parallel write                1,790           1      1   25.7 System I/O
db file sequential read                  49           1     13   12.0   User I/O

Oracle CPU time was 3seconds.
But was the server CPU really idle ?


Statistic                                       Total
-------------------------------- --------------------
AVG_BUSY_TIME                                 721,220
AVG_IDLE_TIME                                 356,777
AVG_SYS_TIME                                    5,531
AVG_USER_TIME                                 715,689
BUSY_TIME                                     721,220
IDLE_TIME                                     356,777
SYS_TIME                                        5,531
USER_TIME                                     715,689
NUM_CPUS                                            1

These statistics show us that CPU %utilisation [including %sys] 
on the server was 66.90% (7212.22seconds out of 10,779.97seconds).
The server was, actually, quite busy -- busy doing something else. Those CPU cycles were consumed by some other application, not being Oracle.

11g Wait Event Histograms

From 11g Oracle introduced Wait Event Histograms in the AWR report. This provides DBAs more granular information in the report for efficient diagonosis.This section splits the event counts into elapsed time buckets so that we can see if there are any suspicious or anomalous response time / frequency patterns. This is a very interesting and informational view in Oracle11g when it comes to analyzing wait time per wait event.

It shows a histogram of the number of waits, total wait time and maximum wait for each event. The bucket sizes or the time intervals in this case, for the histograms are predefined. They cannot be changed. The bucket time intervals are from < 1 ms, < 2 ms, < 4 ms, < 8 ms, < 16 ms, and so on, increasing with the power of 2, till we get to >= 2 ms. The buckets will be populated accordingly, and only when TIMED_STATISTICS is set to TRUE. From a very short report it produces an extremely useful addition to the information about lost time. 

For any event, the less number of buckets the better because the wait times they represent are shorter. The bucket boundaries are pre-set and you cannot change their values. Also, high WAIT_COUNT numbers should be in the buckets with lower WAIT_TIME_MILLI values. If you have it the other way around, then that's the event that needs further investigation.

Another good application of this view is to monitor the SQL*Net message from client event. If the WAIT_COUNT is high for the low-end buckets then it could mean that the application is communicating a lot with the client. This may cause excessive network traffic, which could be reduced with higher ARRAYSIZE or server side processing. On the other hand, if the WAIT_COUNT is high for the high-end buckets then it is more likely due to user lag time.

Below are the screen shots of the various sections of AWR representing Wait Events Statistics.

Wait Event Histogram

  • Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
  • % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
  • % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
  • Ordered by Event (idle events last)
% of Waits
Event Total Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 8 87.5 12.5
Data file init write 7 14.3 28.6 57.1
Disk file operations I/O 587 100.0
LGWR wait for redo copy 12 91.7 8.3
SQL*Net break/reset to client 12 91.7 8.3
SQL*Net message to client 3503 100.0 .0
SQL*Net more data from client 5 100.0
asynch descriptor resize 423 100.0
buffer busy waits 86.7K 24.4 2.7 6.2 13.6 24.5 20.8 7.1 .6
buffer deadlock 2 100.0
control file parallel write 123 3.3 36.6 11.4 3.3 2.4 2.4 40.7
control file sequential read 779 99.7 .1 .1
cr request retry 181 98.3 .6 .6 .6
cursor: pin S 48 43.8 12.5 29.2 8.3 4.2 2.1
db file async I/O submit 52 5.8 1.9 1.9 30.8 59.6
db file parallel read 32 62.5 28.1 3.1 6.3
db file scattered read 195 99.5 .5
db file sequential read 2098 98.0 .1 .1 .2 .4 .4 .8
db file single write 52 30.8 19.2 13.5 5.8 1.9 5.8 23.1
direct path read 733 100.0
direct path sync 1 100.0
direct path write 2 100.0
enq: CF - contention 1 100.0
enq: HW - contention 356 37.9 8.7 20.5 9.0 2.0 5.3 16.6
enq: KO - fast object checkpoint 1 100.0
enq: TX - contention 77 74.0 26.0
enq: US - contention 41 4.9 9.8 24.4 12.2 7.3 17.1 22.0 2.4
free buffer waits 205 .5 1.0 .5 86.3 3.9 5.4 2.4
latch: In memory undo latch 5 100.0
latch: cache buffers chains 195 55.4 16.4 10.3 3.6 4.1 9.7 .5
latch: call allocation 1 100.0
latch: enqueue hash chains 11 81.8 18.2
latch: shared pool 3 100.0
library cache lock 1 100.0
library cache: mutex X 2 50.0 50.0
lock deadlock retry 3 100.0
log buffer space 93 1.1 2.2 3.2 88.2 5.4
log file parallel write 1366 69.0 2.8 2.3 1.8 2.7 4.1 16.5 .7
log file sequential read 10 100.0
log file single write 10 50.0 10.0 40.0
log file switch (checkpoint incomplete) 3 33.3 66.7
log file switch (private strand flush incomplete) 10 60.0 40.0
log file switch completion 7 28.6 57.1 14.3
log file sync 395 19.2 6.8 4.1 5.3 5.6 6.3 48.1 4.6
os thread startup 7 71.4 28.6
rdbms ipc reply 31 93.5 3.2 3.2
reliable message 32 87.5 6.3 3.1 3.1
row cache lock 28 3.6 28.6 28.6 21.4 17.9
DIAG idle wait 275 100.0
SQL*Net message from client 3742 79.3 12.8 3.1 1.6 1.0 .7 .5 .9
Space Manager: slave idle wait 19 5.3 94.7
Streams AQ: qmn coordinator idle wait 10 40.0 10.0 50.0
Streams AQ: qmn slave idle wait 5 100.0
Streams AQ: waiting for messages in the queue 27 100.0
class slave wait 1 100.0
dispatcher timer 3 100.0
jobq slave wait 396 100.0
pmon timer 67 26.9 1.5 3.0 68.7
rdbms ipc message 1101 17.2 4.9 1.2 3.9 5.1 2.7 16.0 49.0
shared server idle wait 5 100.0
smon timer 1 100.0
wait for unread message on broadcast channel 139 .7 .7 98.6

Wait Event Histogram Detail (64 msec to 2 sec)

  • Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
  • Units for % of Total Waits: ms is milliseconds s is 1024 milliseconds (approximately 1 second)
  • % of Total Waits: total waits for all wait classes, including Idle
  • % of Total Waits: value of .0 indicates value was <.05%; value of null is truly 0
  • Ordered by Event (only non-idle events are displayed)
% of Total Waits
Event Waits 64ms to 2s <32ms <64ms <1/8s <1/4s <1/2s <1s <2s >=2s
ADR block file read 1 87.5 12.5
Data file init write 4 42.9 14.3 14.3 14.3 14.3
buffer busy waits 6525 92.3 3.7 .8 .6 .6 1.2 .5 .2
control file parallel write 50 59.3 8.1 10.6 12.2 4.9 4.9
control file sequential read 1 99.9 .1
db file async I/O submit 37 9.6 1.9 1.9 7.7 7.7 11.5 40.4 19.2
db file sequential read 16 99.2 .6 .1 .0 .0
db file single write 12 76.9 13.5 1.9 3.8 1.9 1.9
enq: CF - contention 1 100.0
enq: HW - contention 59 83.4 .3 6.2 10.1
enq: US - contention 10 75.6 4.9 17.1 2.4
free buffer waits 16 92.2 .5 .5 1.0 3.4 2.4
latch: cache buffers chains 1 99.5 .5
log buffer space 86 6.5 5.4 18.3 12.9 24.7 26.9 4.3 1.1
log file parallel write 235 82.7 4.0 4.4 3.4 2.0 2.7 .7 .1
log file single write 4 60.0 10.0 20.0 10.0
log file switch (checkpoint incomplete) 1 33.3 66.7
log file switch (private strand flush incomplete) 8 10.0 20.0 30.0 20.0 20.0
log file switch completion 5 28.6 14.3 14.3 28.6 14.3
log file sync 206 47.3 10.4 9.4 10.6 9.4 8.4 4.1 .5
os thread startup 2 71.4 28.6
reliable message 1 96.9 3.1
row cache lock 5 82.1 17.9

Wait Event Histogram Detail (4 sec to 2 min)

  • Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
  • Units for % of Total Waits: s is 1024 milliseconds (approximately 1 second) m is 64*1024 milliseconds (approximately 67 seconds or 1.1 minutes)
  • % of Total Waits: total waits for all wait classes, including Idle
  • % of Total Waits: value of .0 indicates value was <.05%; value of null is truly 0
  • Ordered by Event (only non-idle events are displayed)
% of Total Waits
Event Waits 4s to 2m <2s <4s <8s <16s <32s < 1m < 2m >=2m
buffer busy waits 156 99.8 .2
db file async I/O submit 10 80.8 19.2
enq: KO - fast object checkpoint 1 100.0
log buffer space 1 98.9 1.1
log file parallel write 1 99.9 .1
log file switch (checkpoint incomplete) 2 33.3 66.7
log file switch (private strand flush incomplete) 2 80.0 20.0
log file sync 2 99.5 .5

 It is always good if you see the " No data exists " in the 4 min to 1 hr segment and 4 sec to 2 min segment.

CPU in AWR Report tips

Question:  My AWR report is showing CPU as the top entry in the top-5 timed events.

I'm concerned because it is consuming 36% of my total DB Time.
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                                           Avg
                                                          wait   % DB
Event                                 Waits     Time(s)   (ms)   time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
DB CPU                                            6,452          36.1
log file sync                       391,362       5,015     13   28.0 Commit
SQL*Net message from dblink       2,925,160       1,891      1   10.6 Network
db file scattered read            1,023,422         305      0    1.7 User I/O
read by other session               499,400         213      0    1.2 User I/O


How can I tell from the AWR report if I am experiencing a CPU bottleneck?


Answer:  Determining CPU in an AWR report is challenging because you have to look in several areas within the AWR report to get the CPU utilization information.  Just because CPU is the #1 timed foreground event, does not mean that the CPU is pegged at 100%, and it does not necessarily mean that you have a "real:" CPU bottleneck, a case where tasks wait in the runqueue for execution.



Note: Before we begin discussing CPU utilization metrics within an AWR report, it is important to understand that oracle statistics are incomplete because of incomplete instrumentation on the Oracle database.  However, you can get accurate CPU metrics viavmstat, as well as top and glance.

Using the example AWR report that you have provided, we can determine the total amount of available CPU and see how much total CPU processing time was available.  The first thing we need to note is the total elapsed time for the AWR report, and this is at the very beginning of the AWR REPORT":


WORKLOAD REPOSITORY report for

DB Name         DB Id    Instance     Inst Num Startup Time    Release     RAC
------------ ----------- ------------ -------- --------------- ----------- ---
FINONE        1252548811 mydb              1 13-Dec-12 22:10 11.2.0.3.0  NO

Host Name        Platform                         CPUs Cores Sockets Memory(GB)
---------------- -------------------------------- ---- ----- ------- ----------
MYPROD           AIX-Based Systems (64-bit)         24    12              80.00

              Snap Id      Snap Time      Sessions Curs/Sess
            --------- ------------------- -------- ---------
Begin Snap:     25130 15-Dec-12 03:00:39       425      12.2
  End Snap:     25131 15-Dec-12 04:00:45       430      14.7
   Elapsed:               60.09 (mins)
   DB Time:              298.22 (mins)


Note above the relationship between the "Elapsed" and "DB Time" columns.  The report period was only one hour (60 minutes), yet we see 298 for DB Time.  This suggests that the (298/60) that there were only five session active at any given time.
Now that we know the total time for the report, if we look farther down into the "OPERATING SYSTEM STATISTICS" section on the AWR report, there are three metrics that are important for determining CPU utilization:
  • BUSY_TIME- The total amount of time that the CPU was busy for the elapsed time period.  This will exceed the wall-=clock time because we usually have multiple CPU's.

  • IDLE_TIME - The amount of time that the database was idle.

  • NUM_CPUS - The number of processors available to the Oracle database


    Here is an example AWR report section showing the total CPU usage:
Operating System Statistics        
-> *TIME statistic values are diffed.
   All others display actual values.  End Value is displayed if different
-> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name

Statistic                                  Value        End Value
------------------------- ---------------------- ----------------
. . .
BUSY_TIME                                811,048
IDLE_TIME                              7,854,100
IOWAIT_TIME                              613,785
SYS_TIME                                 153,228
USER_TIME                                657,820
LOAD                                           1                2
OS_CPU_WAIT_TIME                         871,600
RSRC_MGR_CPU_WAIT_TIME                         0
VM_IN_BYTES                       26,080,108,536
VM_OUT_BYTES                      14,015,483,904
PHYSICAL_MEMORY_BYTES             85,899,280,384
NUM_CPUS                                      24
 . . . 


Now that we have the salient figures, we can calculate the amount of CPU.  The equation for total available CPU is as follows:

      Available CPU = NUM_CPU's * elapsed_time_bet_snapshots * 60(secs) 

                                =  24 * 60 * 60 = 86,400 seconds

Now we can compare that to the amount of used CPU and see that this database was not CPU-bound.  The used DB CPU (6,452 seconds) was only a small fraction of the total available CPU (86,400 seconds).

How to read AWR reports

The output of the AWR report contains a wealth of information that you can use to tune your database. The output of the AWR report can be divided into the following sections:


Report Header

This section is self explanatory which provides database name, id, instance if RAC , platform information and snap interval. (database workload time duration in review).
This report is for instance number 2 of my RAC environment. So if you need to the analysis on RAC environment, you need to do it separately of all the instances in the RAC to see if all the instances are balanced the way they should be.



DB NameDB IdInstanceInst numStartup TimeReleaseRAC
TestRAC3626203793TestRac2217-Aug-11 19:0811.1.0.6.0YES


Host NamePlatformCPUsCoresSocketsMemory (GB)
TestRACLinux 64-bit for AMD88231.44


Snap IdSnap TimeSessionsCursors/Session
Begin Snap:2856627-Sep-11 01:00:211304.8
End Snap:2856727-Sep-11 02:00:431354.5
Elapsed:60.35 (mins)
DB Time:15.07 (mins)

BeginEnd
Buffer Cache:5,888M5,888MStd Block Size:8K
Shared Pool Size:8,704M8,704MLog Buffer:138,328K




Load Profile
This section provides the snapshot of the database workload occurred during the snapshot interval.


Per SecondPer TransactionPer ExecPer Call
DB Time(s):0.30.10.000.00
DB CPU(s):0.30.10.000.00
Redo size:48,933.619,916.2
Logical reads:1,124.4457.7
Block changes:195.979.7
Physical reads:80.532.8
Physical writes:4.31.8
User calls:141.457.6
Parses:123.250.2
Hard parses:2.20.9
W/A MB processed:1,940,807.0789,918.9
Logons:4.31.7
Executes:127.651.9
Rollbacks:0.00.0
Transactions:2.5



DB time(s):
Its the amount of time oracle has spent performing database user calls. Note it does not include background processes.
DB CPU(s):
Its the amount of CPU time spent on user calls. Same as DB time it does not include background process. The value is in microseconds
Redo size:
 For example, the table below shows that an average transaction generates about 19,000 of redo data along with around 48,000 redo per second.
Logical reads:
Consistent Gets+ DB blocks Gets = Logical reads
Block Changes:
The number of block modified during the sample interval
Physical reads:
No of block request causing I/O operation
Physical writes:
Number of physical writes performed
User calls:
Number of user queries generated
Parses:
The total of all parses; both hard and soft.
Hard Parses: 
The parses requiring a completely new parse of the SQL statement. These consume both latches and shared pool area.
Soft Parses:
Soft parses are not listed but derived by subtracting the hard parses from parses. A soft parse reuses a previous hard parse; hence it consumes far fewer resources.
Sorts:
No of sorts performed
Logons:
No of logons during the interval
Executes:
No of SQL Executes
Transactions
No of transactions per second

---------------------------------------------------------------------------

Load Profile

The load profile provides an at-a-glance look at some specific operational statistics. You can compare these statistics with a baseline snapshot report to determine if database activity is different. Values for these statistics are presented in two formats. The first is the value per second (for example, how much redo was generated per second) and the second is the value per transaction (for example, 1,024 bytes of redo were generated per transaction).
Statistics presented in the load profile include such things as:
  • Redo size - An indication of the amount of DML activity the database is experiencing.
     
  • Logical and physical reads - A measure of how many IO's (Physical and logical) that the database is performing.
     
  • User calls - Indicates how many user calls have occurred during the snapshot period. This value can give you some indication if usage has increased.
     
  • Parses and hard parses - Provides an indication of the efficiency of SQL re-usage.
     
  • Sorts - This number gives you an indication of how much sorting is occurring in the database.
     
  • Logons - Indicates how many logons occurred during the snapshot period.
     
  • Executes - Indicates how many SQL statements were executed during the snapshot period.
     
  • Transactions - Indicates how many transactions occurred during the snapshot period.
Additionally, the load profile section provides the percentage of blocks that were changed per read, the percentage of recursive calls that occurred, the percentage of transactions that were rolled back and the number of rows sorted per sort operation.


Instance Efficiency Percentages (Target 100%)

 These statistics include several buffer related ratios including the buffer hit percentage and the library hit percentage. Also, shared pool memory usage statistics are included in this section.

Instance efficiency should be close to 100 %


Buffer Nowait %:99.99Redo NoWait %:100.00
Buffer Hit %:93.06In-memory Sort %:100.00
Library Hit %:98.67Soft Parse %:98.20
Execute to Parse %:3.40Latch Hit %:99.98
Parse CPU to Parse Elapsd %:0.01% Non-Parse CPU:96.21

Execute to Parse % and Parse CPU to Parse Elapsd %:

If the the value are low like in the above case of 3.40 and 0.01 means that there could be a parsing problem. You may need to look at bind variable issues or shared pool sizing issue.

Redo NoWait%:

Usually this stats is 99 or greater

In-memory Sort %:
This can tell you how efficient is you sort_area_size, hash_area_size or pga_aggrigate_target are. If you dont have adequate sizes of sort,hash and pga parameters, then you in-memory sort per cent will go down

Soft parse %:
with 98.20 % for the soft parse meaning that about 1.72 % (100 -soft parse) is happening for hard parsing. You might want to look at you bind variables issues.

Latch Hit %:
should be close to 100.

% Non-Parse CPU:
Most of our statements were already parsed so we weren't doing a lot of re parsing. Re parsing is high on CPU and should be avoided.
-------------------------------------------------------------------------------------------------------------------------------

Shared Pool Statistics


BeginEnd
Memory Usage %:73.8675.42
% SQL with executions>1:92.6193.44
% Memory for SQL w/exec>1:94.3394.98



Memory Usage % is the shared pool usage. So here we have use 73.86 per cent of our shared pool and out of that almost 94 percent is being re-used. if Memory Usage % is too large like 90 % it could mean that your shared pool is tool small and if the percent is in 50 for example then this could mean that you shared pool is too large

---------------------------------------------------------------------------------------------------------------------------------------
Top 5 Timed Foreground Events

This section provides insight into what events the Oracle database is spending most of it's time on (see wait events). Each wait event is listed, along with the number of waits, the time waited (in seconds), the average wait per event (in microseconds) and the associated wait class

    EventWaitsTime(s)Avg wait (ms)% DB timeWait Class
    DB CPU1,019112.73
    log file sync25,6424324.73Commit
    db file scattered read3,06440134.43User I/O
    library cache pin136,2672702.98Concurrency
    db file sequential read7,6082432.71User I/O

    its critical to look into this section. If you turn off the statistic parameter, then the Time(s) wont appear. Wait analysis should be done with respect to Time(s) as there could be million of waits but if that happens for a second or so then who cares. Therefore, time is very important component.

    So you have several different types of waits. So you may see the different waits on your AWR report. So lets discuss the most common waits.


    • df file type waits:


    db file sequential read:
    Is the wait that comes from the physical side of the database. it related to memory starvation and non selective index use. sequential read is an index read followed by table read because it is doing index lookups which tells exactly which block to go to
    db file scattered read:
    caused due to full table scans may be because of insufficient indexes or un-avilablity of updated statistics
    direct Path writes:
    You wont see them unless you are doing some appends or data loads
    direct Path reads:
    could happen if you are doing a lot of parallel query activity
    db file parallel writes / read: 
    if you are doing a lot of partition activity then expect to see that wait even. it could be a table or index partition
    db file single write:
    if you see this event than probably you have a lot of data files in your database.
    direct path read temp or direct path write temp:
    this wait event shows Temp file activity (sort,hashes,temp tables, bitmap)
    check pga parameter or sort area or hash area parameters. You might want to increase them


    • buffer type waits

    so what's going on in your memory
    latch: cache buffer chains:
    check hot objects
    free buffer waits:
    insufficient buffers, process holding buffers too long or i/o subsystem is over loaded. Also check you db writes may be getting clogged up.
    buffer busy waits:
    see what is causing them further along in report. most of the time its data block related.
    gc buffer busy:
    its in the RAC environment. caused may be because of not enough memory on your nodes,overloaded interconnect. Also look RAC specific section of the report latch:
    cache buffers lru chain – Freelist issues, hot blocks latch: cache buffer handles – Freelist issues, hot blocks
    buffer busy - See what is causing them further along in report
    no free buffers – Insufficient buffers, dbwr contention


    • Log Type Waits
    log file parallel write – Look for log file contention
    log buffer space – Look at increasing log buffer size
    log file switch (checkpoint incomplete) – May indicate excessive db files or slow IO subsystem
    log file switch (archiving needed) – Indicates archive files are written too slowly
    log file switch completion – May need more log files per
    log file sync – Could indicate excessive commits



    • GC Events
    gccr multi block request – Full table or index scans
    gc current multi block request – Full table or index scans
    gccr block 2-way – Blocks are busy in another instance, check for block level contention or hot blocks
    gccr block 3-way – Blocks are busy in another instance, check for block level contention or hot blocks
    gccr block busy – Blocks are busy in another instance, check for block level contention or hot blocks
    gccr block congested – cr block congestion, check for hot blocks or busy interconnect
    gccr block lost – Indicates interconnect issues and contention
    gc current block 2-way – Blocks are busy in another instance, check for block level contention or hot blocks
    gc current block 3-way – Blocks are busy in another instance, check for block level contention or hot blocks
    gc current block busy – Block is already involved in GC operation, shows hot blocks or congestion
    gc current block congested – current block congestion, check for hot blocks or busy interconnect
    gc current block lost - Indicates interconnect issues and contention



    • Undo Events
    undo segment extension – If excessive, tune undo
    latch: In memory undo latch – If excessive could be bug, check for your version, may have to turn off in memory undo
    wait for a undo record – Usually only during recovery of large transactions, look at turning off parallel undo recovery.
    • What Next?
    Determine wait events of concern
    Drill down to specific sections of report for deeper analysis
    Use custom scripts, ADDM and Ash to investigate issues

    ======================================================================
    ======================================================================

    RAC Statistics

    If you are running on a RAC cluster, then the AWRRPT.SQL report will provide various RAC statistics including statistics on the number of RAC instances, as well as global cache and enqueue related performance statistics. Here is an example of the RAC statistics part of the report:
    RAC Statistics  DB/Inst: A109/a1092  Snaps: 2009-2010
    
                                    Begin   End
                                    ----- -----
               Number of Instances:     2     2
    
    
    Global Cache Load Profile
    ~~~~~~~~~~~~~~~~~~~~~~~~~                  Per Second       Per Transaction
                                          ---------------       ---------------
      Global Cache blocks received:                  0.11                  0.52
        Global Cache blocks served:                  0.14                  0.68
         GCS/GES messages received:                  0.88                  4.23
             GCS/GES messages sent:                  0.85                  4.12
                DBWR Fusion writes:                  0.01                  0.04
     Estd Interconnect traffic (KB)                  2.31
    
    
    Global Cache Efficiency Percentages (Target local+remote 100%)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Buffer access -  local cache %:   99.47
    Buffer access - remote cache %:    0.53
    Buffer access -         disk %:    0.00
    
    
    Global Cache and Enqueue Services - Workload Characteristics
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                         Avg global enqueue get time (ms):      0.0
    
              Avg global cache cr block receive time (ms):      0.2
         Avg global cache current block receive time (ms):      0.3
    
                Avg global cache cr block build time (ms):      0.0
                 Avg global cache cr block send time (ms):      0.0
          Global cache log flushes for cr blocks served %:      1.8
                Avg global cache cr block flush time (ms):      4.0
    
             Avg global cache current block pin time (ms):      0.0
            Avg global cache current block send time (ms):      0.1
     Global cache log flushes for current blocks served %:      0.4
           Avg global cache current block flush time (ms):      0.0
    
    Global Cache and Enqueue Services - Messaging Statistics
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                         Avg message sent queue time (ms): ########
                 Avg message sent queue time on ksxp (ms):      0.1
                     Avg message received queue time (ms):      4.6
                        Avg GCS message process time (ms):      0.0
                        Avg GES message process time (ms):      0.0
    
                                % of direct sent messages:    45.26
                              % of indirect sent messages:    31.59
                            % of flow controlled messages:    23.15
              -------------------------------------------------------------

    Time Model Statistics

    Oracle Database 10g time model related statistics are presented next. The time model allows you to see a summary of where the database is spending it's time. The report will present the various time related statistic (such as DB CPU) and how much total time was spent in the mode of operation represented by that statistic. Here is an example of the time model statistic report where we see that we spent 36.2 seconds on DB CPU time, which was a total of 60.4% of the total DB time. Note that this is a two node RAC system, so the total percentage of overall time available is 200%, not 100%.
    Time Model Statistics                   DB/Inst: A109/a1092  Snaps: 2009-2010
    -> Total time in database user-calls (DB Time): 5.5s
    -> Statistics including the word "background" measure background process
       time, and so do not contribute to the DB time statistic
    -> Ordered by % or DB time desc, Statistic name
    
    Statistic Name                                       Time (s) % of DB Time
    ------------------------------------------ ------------------ ------------
    sql execute elapsed time                                  4.5         82.8
    DB CPU                                                    3.5         64.4
    connection management call elapsed time                   0.1          1.6
    parse time elapsed                                        0.1          1.3
    PL/SQL execution elapsed time                             0.0           .9
    hard parse elapsed time                                   0.0           .3
    sequence load elapsed time                                0.0           .1
    repeated bind elapsed time                                0.0           .0
    DB time                                                   5.5          N/A
    background elapsed time                                  33.0          N/A
    background cpu time                                       9.7          N/A
              -------------------------------------------------------------

    Wait class and Wait Event Statistics

    Closely associated with the time model section of the report are the wait class and wait event statistics sections. Within Oracle, the duration of a large number of operations (e.g. Writing to disk or to the control file) is metered. These are known as wait events, because each of these operations requires the system to wait for the event to complete. Thus, the execution of some database operation (e.g. a SQL query) will have a number of wait events associated with it. We can try to determine which wait events are causing us problems by looking at the wait classes and the wait event reports generated from AWR.
    Wait classes define "buckets" that allow for summation of various wait times. Each wait event is assigned to one of these buckets (for example System I/O or User I/O). These buckets allow one to quickly determine which subsystem is likely suspect in performance problems (e.g. the network, or the cluster). Here is an example of the wait class report section:
    Wait Class                               DB/Inst: A109/a1092  Snaps: 2009-2010
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc
    
                                                                      Avg
                                           %Time       Total Wait    wait     Waits
    Wait Class                      Waits  -outs         Time (s)    (ms)      /txn
    -------------------- ---------------- ------ ---------------- ------- ---------
    System I/O                      8,142     .0               25       3      10.9
    Other                         439,596   99.6                3       0     589.3
    User I/O                          112     .0                0       3       0.2
    Cluster                           443     .0                0       0       0.6
    Concurrency                       216     .0                0       0       0.3
    Commit                             16     .0                0       2       0.0
    Network                         3,526     .0                0       0       4.7
    Application                        13     .0                0       0       0.0
              -------------------------------------------------------------
    In this report the system I/O wait class has the largest number of waits (total of 25 seconds) and an average wait of 3 milliseconds.
    Wait events are normal occurrences, but if a particular sub-system is having a problem performing (e.g. the disk sub-system) this fact will appear in the form of one or more wait events with an excessive duration. The wait event report then provides some insight into the detailed wait events. Here is an example of the wait event report (we have eliminated some of the bulk of this report, because it can get quite long). Note that this section is sorted by wait time (listed in microseconds).
                                                                       Avg
                                                 %Time  Total Wait    wait     Waits
    Event                                 Waits  -outs    Time (s)    (ms)      /txn
    ---------------------------- -------------- ------ ----------- ------- ---------
    control file parallel write           1,220     .0          18      15       1.6
    control file sequential read          6,508     .0           6       1       8.7
    CGS wait for IPC msg                422,253  100.0           1       0     566.0
    change tracking file synchro             60     .0           1      13       0.1
    db file parallel write                  291     .0           0       1       0.4
    db file sequential read                  90     .0           0       4       0.1
    reliable message                        136     .0           0       1       0.2
    log file parallel write                 106     .0           0       2       0.1
    lms flush message acks                    1     .0           0      60       0.0
    gc current block 2-way                  200     .0           0       0       0.3
    change tracking file synchro             59     .0           0       1       0.1
    In this example our control file parallel write waits (which occurs during writes to the control file) are taking up 18 seconds total, with an average wait of 15 milliseconds per wait. Additionally we can see that we have 1.6 waits per transaction (or 15ms * 1.6 per transaction = 24ms).

    Operating System Statistics

    This part of the report provides some basic insight into OS performance, and OS configuration too. This report may vary depending on the OS platform that your database is running on. Here is an example from a Linux system:
    Statistic                                       Total
    -------------------------------- --------------------
    BUSY_TIME                                     128,749
    IDLE_TIME                                   1,314,287
    IOWAIT_TIME                                    18,394
    NICE_TIME                                          54
    SYS_TIME                                       31,633
    USER_TIME                                      96,586
    LOAD                                                0
    RSRC_MGR_CPU_WAIT_TIME                              0
    PHYSICAL_MEMORY_BYTES                       3,349,528
    NUM_CPUS                                            4
    In this example output, for example, we have 4 CPU's on the box.

    SQL In Need of Tuning

    Next in the report we find several different reports that present SQL statements that might be improved by tuning. There are a number of different reports that sort offending SQL statements by the following criteria:
    • Elapsed time
       
    • CPU time
       
    • Buffer gets
       
    • Physical reads
       
    • Executions
       
    • Parse calls
       
    • Sharable memory
       
    • Version count
       
    • Cluster wait time
    While these reports might not help tune specific application problems, they can help you find more systemic SQL problems that you might not find when tuning a specific application module. Here is an example of the Buffer gets report:
                                    Gets              CPU     Elapsed
      Buffer Gets   Executions    per Exec   %Total Time (s)  Time (s)    SQL Id
    -------------- ------------ ------------ ------ -------- --------- -------------
             2,163            7        309.0    3.0     0.03      0.04 c7sn076yz7030
    select smontabv.cnt, smontab.time_mp, smontab.scn, smontab.num_mappings, smon
    tab.tim_scn_map, smontab.orig_thread from smon_scn_time smontab, (sel
    ect max(scn) scnmax, count(*)+sum(NVL2(TIM_SCN_MAP,NUM_MAPPINGS,
    0)) cnt from smon_scn_time where thread=0) smontabv where smon
    
             1,442          721          2.0    2.0     0.05      0.05 6ssrk2dqj7jbx
    select job, nvl2(last_date, 1, 0) from sys.job$ where (((:1 <= next_date) and (n
    ext_date <= :2)) or ((last_date is null) and (next_date < :3))) and (field1
    = :4 or (field1 = 0 and 'Y' = :5)) and (this_date is null) order by next_date, j
    ob
    
             1,348            1      1,348.0    1.9     0.04      0.04 bv1djzzmk9bv6
    Module: TOAD 9.0.0.160
    Select table_name from DBA_TABLES where owner = 'CDOL2_01' order by 1
    
             1,227            1      1,227.0    1.7     0.07      0.08 d92h3rjp0y217
    begin prvt_hdm.auto_execute( :db_id, :inst_id, :end_snap ); end;
    
               896            4        224.0    1.2     0.03      0.03 6hszmvz1wjhbt
    Module: TOAD 9.0.0.160
    Select distinct Cons.constraint_name, cons.status, cons.table_name, cons.constra
    int_type ,cons.last_change from sys.user_constraints cons where 1=1 a
    nd cons.status='DISABLED'
    In this report we find a SQL statement that seems to be churning through 309 buffers per execution. While the execution times are not terrible we might want to look closer into the SQL statement and try to see if we could tune it (in fact this is Oracle issued SQL that we would not tune anyway).

    Instance Activity Stats

    This section provides us with a number of various statistics (such as, how many DBWR Checkpoints occurred, or how many consistent gets occurred during the snapshot). Here is a partial example of the report:
    Statistic                                     Total     per Second     per Trans
    -------------------------------- ------------------ -------------- -------------
    consistent changes                                9            0.0           0.0
    consistent gets                              70,445           19.5          94.4
    consistent gets - examination                 8,728            2.4          11.7
    consistent gets direct                            0            0.0           0.0
    consistent gets from cache                   70,445           19.5          94.4
    cursor authentications                            2            0.0           0.0
    data blocks consistent reads - u                  5            0.0           0.0
    db block changes                              1,809            0.5           2.4
    db block gets                                 2,197            0.6           3.0
    db block gets direct                              0            0.0           0.0
    db block gets from cache                      2,033            0.6           2.7

    Tablespace and Data File IO Stats

    The tablespace and data file IO stats report provides information on tablespace IO performance. From this report you can determine if the tablespace datafiles are suffering from sub-standard performance in terms of IO response from the disk sub-system. Here is a partial example of the tablespace report:
    Tablespace
    ------------------------------
                     Av      Av     Av                       Av     Buffer Av Buf
             Reads Reads/s Rd(ms) Blks/Rd       Writes Writes/s      Waits Wt(ms)
    -------------- ------- ------ ------- ------------ -------- ---------- ------
    SYSAUX
                 1       0    0.0     1.0          159        0         13    0.8
    UNDOTBS2
                 1       0   10.0     1.0           98        0          0    0.0
    SYSTEM
                 1       0   10.0     1.0           46        0          0    0.0
    AUD
                 1       0    0.0     1.0            1        0          0    0.0
    CDOL2_INDEX
                 1       0   10.0     1.0            1        0          0    0.0
    CDOL_DATA
                 1       0   10.0     1.0            1        0          0    0.0
    DBA_DEF
                 1       0   10.0     1.0            1        0          0    0.0
    UNDOTBS1
                 1       0   10.0     1.0            1        0          0    0.0
    USERS
                 1       0   10.0     1.0            1        0          0    0.0
    USER_DEF
                 1       0   10.0     1.0            1        0          0    0.0
    If the tablespace IO report seems to indicate a tablespace has IO problems, we can then use the file IO stat report allows us to drill into the datafiles of the tablespace in question and determine what the problem might be. Here is an example of the File IO stat report:
    Tablespace               Filename
    ------------------------ ----------------------------------------------------
                     Av      Av     Av                       Av     Buffer Av Buf
             Reads Reads/s Rd(ms) Blks/Rd       Writes Writes/s      Waits Wt(ms)
    -------------- ------- ------ ------- ------------ -------- ---------- ------
    AUD                      +ASM01/a109/datafile/aud.296.604081931
                 1       0    0.0     1.0            1        0          0    0.0
    CDOL2_INDEX              +ASM01/a109/datafile/cdol2_index_001.dbf
                 1       0   10.0     1.0            1        0          0    0.0
    CDOL_DATA                +ASM01/a109/datafile/cdol_data_001.dbf
                 1       0   10.0     1.0            1        0          0    0.0
    DBA_DEF                  +ASM01/a109/datafile/dba_def.294.604081931
                 1       0   10.0     1.0            1        0          0    0.0
    SYSAUX                   +ASM01/a109/datafile/sysaux.299.604081927
                 1       0    0.0     1.0          159        0         13    0.8
    SYSTEM                   +ASM01/a109/datafile/system.301.604081919
                 1       0   10.0     1.0           46        0          0    0.0
    UNDOTBS1                 +ASM01/a109/datafile/undotbs1.300.604081925
                 1       0   10.0     1.0            1        0          0    0.0
    UNDOTBS2                 +ASM01/a109/datafile/undotbs2.292.604081931
                 1       0   10.0     1.0           98        0          0    0.0
    USERS                    +ASM01/a109/datafile/users.303.604081933
                 1       0   10.0     1.0            1        0          0    0.0
    USER_DEF                 +ASM01/a109/datafile/user_def.291.604081933
                 1       0   10.0     1.0            1        0          0    0.0
              -------------------------------------------------------------

    Buffer Pool Statistics

    The buffer pool statistics report follows. It provides a summary of the buffer pool configuration and usage statistics as seen in this example:
                                                                Free Writ     Buffer
         Number of Pool         Buffer     Physical    Physical Buff Comp       Busy
    P      Buffers Hit%           Gets        Reads      Writes Wait Wait      Waits
    --- ---------- ---- -------------- ------------ ----------- ---- ---- ----------
    D       64,548  100         72,465            0         355    0    0         13
              -------------------------------------------------------------
    In this case, we have a database where all the buffer pool requests came out of the buffer pool and no physical reads were required. We also see a few (probably very insignificant in our case) buffer busy waits.

    Instance Recovery Stats

    The instance recovery stats report provides information related to instance recovery. By analyzing this report, you can determine roughly how long your database would have required to perform crash recovery during the reporting period. Here is an example of this report:
    -> B: Begin snapshot,  E: End snapshot
    
      Targt  Estd                                  Log File Log Ckpt     Log Ckpt
      MTTR   MTTR   Recovery  Actual    Target       Size    Timeout     Interval
       (s)    (s)   Estd IOs Redo Blks Redo Blks  Redo Blks Redo Blks   Redo Blks
    - ----- ----- ---------- --------- --------- ---------- --------- ------------
    B     0    19        196       575       183      92160       183          N/A
    E     0    19        186       258        96      92160        96          N/A
              -------------------------------------------------------------

    Buffer Pool Advisory

    The buffer pool advisory report answers the question, how big should you make your database buffer cache. It provides an extrapolation of the benefit or detriment that would result if you added or removed memory from the database buffer cache. These estimates are based on the current size of the buffer cache and the number of logical and physical IO's encountered during the reporting point. This report can be very helpful in "rightsizing" your buffer cache. Here is an example of the output of this report:
                                            Est
                                           Phys
        Size for   Size      Buffers for   Read          Estimated
    P    Est (M) Factor         Estimate Factor     Physical Reads
    --- -------- ------ ---------------- ------ ------------------
    D         48     .1            5,868    4.9            803,496
    D         96     .2           11,736    4.0            669,078
    D        144     .3           17,604    3.3            550,831
    D        192     .4           23,472    2.8            462,645
    D        240     .5           29,340    2.3            379,106
    D        288     .5           35,208    1.8            305,342
    D        336     .6           41,076    1.4            238,729
    D        384     .7           46,944    1.2            200,012
    D        432     .8           52,812    1.1            183,694
    D        480     .9           58,680    1.0            172,961
    D        528    1.0           64,548    1.0            165,649
    D        576    1.1           70,416    1.0            161,771
    D        624    1.2           76,284    1.0            159,728
    D        672    1.3           82,152    1.0            158,502
    D        720    1.4           88,020    1.0            157,723
    D        768    1.5           93,888    0.9            157,124
    D        816    1.5           99,756    0.9            156,874
    D        864    1.6          105,624    0.9            156,525
    D        912    1.7          111,492    0.9            156,393
    D        960    1.8          117,360    0.9            155,388
              -------------------------------------------------------------
    In this example we currently have 528GB allocated to the SGA (represented by the size factor column with a value of 1.0. It appears that if we were to reduce the memory allocated to the SGA to half of the size of the current SGA (freeing the memory to the OS for other processes) we would incur an increase of about 1.8 times the number of physical IO's in the process.

    PGA Reports

    The PGA reports provide some insight into the health of the PGA. The PGA Aggr Target Stats report provides information on the configuration of the PGA Aggregate Target parameter during the reporting period.
    The PGA Aggregate Target Histogram report provides information on the size of various operations (e.g. sorts). It will indicate if PGA sort operations occurred completely in memory, or if some of those operations were written out to disk.
    Finally the PGA Memory Advisor, much like the buffer pool advisory report, provides some insight into how to properly size your PGA via the PGA_AGGREGATE_TARGET database parameter. The PGA Memory Advisor report is shown here:
                                           Estd Extra    Estd PGA   Estd PGA
    PGA Target    Size           W/A MB   W/A MB Read/      Cache  Overalloc
      Est (MB)   Factr        Processed Written to Disk     Hit %      Count
    ---------- ------- ---------------- ---------------- -------- ----------
            44     0.1        289,899.2          7,844.9     97.0      1,124
            88     0.3        289,899.2          7,576.9     97.0      1,073
           176     0.5        289,899.2              3.3    100.0          0
           263     0.8        289,899.2              3.3    100.0          0
           351     1.0        289,899.2              3.3    100.0          0
           421     1.2        289,899.2              0.0    100.0          0
           491     1.4        289,899.2              0.0    100.0          0
           562     1.6        289,899.2              0.0    100.0          0
           632     1.8        289,899.2              0.0    100.0          0
           702     2.0        289,899.2              0.0    100.0          0
         1,053     3.0        289,899.2              0.0    100.0          0
         1,404     4.0        289,899.2              0.0    100.0          0
         2,106     6.0        289,899.2              0.0    100.0          0
         2,808     8.0        289,899.2              0.0    100.0          0
              -------------------------------------------------------------

    Shared Pool Advisory

    The shared pool advisory report provides assistance in right sizing the Oracle shared pool. Much like the PGA Memory Advisor or the Buffer Pool advisory report, it provides some insight into what would happen should you add or remove memory from the shared pool. This can help you reclaim much needed memory if you have over allocated the shared pool, and can significantly improve performance if you have not allocated enough memory to the shared pool. Here is an example of the shared pool advisory report:
                                            Est LC Est LC  Est LC Est LC
        Shared    SP   Est LC                 Time   Time    Load   Load      Est LC
          Pool  Size     Size       Est LC   Saved  Saved    Time   Time         Mem
       Size(M) Factr      (M)      Mem Obj     (s)  Factr     (s)  Factr    Obj Hits
    ---------- ----- -------- ------------ ------- ------ ------- ------ -----------
           192    .4       54        3,044 #######     .8 #######  382.1  22,444,274
           240    .5       92        5,495 #######     .9 #######  223.7  22,502,102
           288    .6      139        8,122 #######     .9  53,711  102.5  22,541,782
           336    .7      186       12,988 #######    1.0  17,597   33.6  22,562,084
           384    .8      233       17,422 #######    1.0   7,368   14.1  22,569,402
           432    .9      280       23,906 #######    1.0   3,553    6.8  22,571,902
           480   1.0      327       28,605 #######    1.0     524    1.0  22,573,396
           528   1.1      374       35,282 #######    1.0       1     .0  22,574,164
           576   1.2      421       40,835 #######    1.0       1     .0  22,574,675
           624   1.3      468       46,682 #######    1.0       1     .0  22,575,055
           672   1.4      515       52,252 #######    1.0       1     .0  22,575,256
           720   1.5      562       58,181 #######    1.0       1     .0  22,575,422
           768   1.6      609       64,380 #######    1.0       1     .0  22,575,545
           816   1.7      656       69,832 #######    1.0       1     .0  22,575,620
           864   1.8      703       75,168 #######    1.0       1     .0  22,575,668
           912   1.9      750       78,993 #######    1.0       1     .0  22,575,695
           960   2.0      797       82,209 #######    1.0       1     .0  22,575,719
              -------------------------------------------------------------

    SGA Target Advisory

    The SGA target advisory report is somewhat of a summation of all the advisory reports previously presented in the AWR report. It helps you determine the impact of changing the settings of the SGA target size in terms of overall database performance. The report uses a value called DB Time as a measure of the increase or decrease in performance relative to the memory change made. Also the report will summarize an estimate of physical reads associated with the listed setting for the SGA. Here is an example of the SGA target advisory report:
    SGA Target   SGA Size       Est DB     Est Physical
      Size (M)     Factor     Time (s)            Reads
    ---------- ---------- ------------ ----------------
           528        0.5       25,595          769,539
           792        0.8       20,053          443,095
         1,056        1.0       18,443          165,649
         1,320        1.3       18,354          150,476
         1,584        1.5       18,345          148,819
         1,848        1.8       18,345          148,819
         2,112        2.0       18,345          148,819
    In this example, our SGA Target size is currently set at 1056MB. We can see from this report that if we increased the SGA target size to 2112MB, we would see almost no performance improvement (about a 98 second improvement overall). In this case, we may determine that adding so much memory to the database is not cost effective, and that the memory can be better used elsewhere.

    Memory Advisory

    Memory advisory reports for the streams pool and the java pool also appear in the report (assuming you are using the streams pool). These reports take on the same general format as the other memory advisor reports.

    Buffer Wait Statistics

    The buffer wait statistics report helps you drill down on specific buffer wait events, and where the waits are occurring. In the following report we find that the 13 buffer busy waits we saw in the buffer pool statistics report earlier are attributed to data block waits. We might then want to pursue tuning remedies to these waits if the waits are significant enough. Here is an example of the buffer wait statistics report:
    Class                    Waits Total Wait Time (s)  Avg Time (ms)
    ------------------ ----------- ------------------- --------------
    data block                  13                   0              1

    Enqueue Activity

    The Enqueue activity report provides information on enqueues (higher level Oracle locking) that occur. As with other reports, if you see high levels of wait times in these reports, you might dig further into the nature of the enqueue and determine the cause of the delays. Here is an example of this report section:
    Enqueue Type (Request Reason)
    ------------------------------------------------------------------------------
        Requests    Succ Gets Failed Gets       Waits  Wt Time (s) Av Wt Time(ms)
    ------------ ------------ ----------- ----------- ------------ --------------
    PS-PX Process Reservation
             386          358          28         116            0            .43
    US-Undo Segment
             276          276           0         228            0            .18
    TT-Tablespace
              90           90           0          42            0            .71
    WF-AWR Flush
              12           12           0           7            0           1.43
    MW-MWIN Schedule
               2            2           0           2            0           5.00
    TA-Instance Undo
              12           12           0          12            0            .00
    UL-User-defined
               7            7           0           7            0            .00
    CF-Controlfile Transaction
           5,737        5,737           0           5            0            .00

    Undo Segment Summary

    The undo segment summary report provides basic information on the performance of undo tablespaces.

    Latch Activity

    The latch activity report provides information on Oracle's low level locking mechanism called a latch. From this report you can determine if Oracle is suffering from latching problems, and if so, which latches are causing the greates amount of contention on the system. Here is a partial example of the latch activity report (it is quite long):
                                               Pct    Avg   Wait                 Pct
                                        Get    Get   Slps   Time       NoWait NoWait
    Latch Name                     Requests   Miss  /Miss    (s)     Requests   Miss
    ------------------------ -------------- ------ ------ ------ ------------ ------
    ASM allocation                      122    0.0    N/A      0            0    N/A
    ASM map headers                      60    0.0    N/A      0            0    N/A
    ASM map load waiting lis             11    0.0    N/A      0            0    N/A
    ASM map operation freeli             30    0.0    N/A      0            0    N/A
    ASM map operation hash t         45,056    0.0    N/A      0            0    N/A
    ASM network background l          1,653    0.0    N/A      0            0    N/A
    AWR Alerted Metric Eleme         14,330    0.0    N/A      0            0    N/A
    Consistent RBA                      107    0.0    N/A      0            0    N/A
    FAL request queue                    75    0.0    N/A      0            0    N/A
    FAL subheap alocation                75    0.0    N/A      0            0    N/A
    FIB s.o chain latch                  14    0.0    N/A      0            0    N/A
    FOB s.o list latch                   93    0.0    N/A      0            0    N/A
    JS broadcast add buf lat            826    0.0    N/A      0            0    N/A
    JS broadcast drop buf la            826    0.0    N/A      0            0    N/A
    In this example our database does not seem to be experiencing any major latch problems, as the wait times on the latches are 0, and our get miss pct (Pct Get Miss) is 0 also.
    There is also a latch sleep breakdown report which provides some additional detail if a latch is being constantly moved into the sleep cycle, which can cause additional performance issues.
    The latch miss sources report provides a list of latches that encountered sleep conditions. This report can be of further assistance when trying to analyze which latches are causing problems with your database.

    Segments by Logical Reads and Segments by Physical Reads

    The segments by logical reads and segments by physical reads reports provide information on the database segments (tables, indexes) that are receiving the largest number of logical or physical reads. These reports can help you find objects that are "hot" objects in the database. You may want to review the objects and determine why they are hot, and if there are any tuning opportunities available on those objects (e.g. partitioning), or on SQL accessing those objects.
    For example, if an object is showing up on the physical reads report, it may be that an index is needed on that object. Here is an example of the segments by logical reads report:
    Segments by Logical Reads               DB/Inst: A109/a1092  Snaps: 2009-2010
    -> Total Logical Reads:          72,642
    -> Captured Segments account for   96.1% of Total
    
               Tablespace                      Subobject  Obj.       Logical
    Owner         Name    Object Name            Name     Type         Reads  %Total
    ---------- ---------- -------------------- ---------- ----- ------------ -------
    SYS        SYSAUX     SYS_IOT_TOP_8813                INDEX       52,192   71.85
    SYS        SYSTEM     SMON_SCN_TIME                   TABLE        4,704    6.48
    SYS        SYSTEM     I_JOB_NEXT                      INDEX        2,432    3.35
    SYS        SYSTEM     OBJ$                            TABLE        1,344    1.85
    SYS        SYSTEM     TAB$                            TABLE        1,008    1.39
              -------------------------------------------------------------

    Additional Reports

    Several segment related reports appear providing information on:
    • Segments with ITL waits
       
    • Segments with Row lock waits
       
    • Segments with buffer busy waits
       
    • Segments with global cache buffer waits
       
    • Segments with CR Blocks received
       
    • Segments with current blocks received
    These reports help provide more detailed information on specific segments that might be experiencing performance problems.
    The dictionary cache and library cache statistics reports provide performance information on the various areas in the data dictionary cache and the library cache.
    The process memory summary, SGA memory summary, and the SGA breakdown difference reports provide summary information on how memory allocated to the database is allocated amongst the various components. Other memory summary reports may occur if you have certain optional components installed (such as streams).
    The database parameter summary report provides a summary of the setting of all the database parameters during the snapshot report. If the database parameters changed during the period of the report, then the old and new parameters will appear on the report.