RMOUG 2012 training days

February 12, 2012, 11:44 pm

≫ Next: Fast Analytics of AWR Top Events

≪ Previous: SQLTXPLAIN quick tips and tricks and DB Optimizer VST

In the next few days I’ll be at RMOUG 2012 training days! and I’ll be again presenting the topic so dear to my heart. Mining the AWR I’ve updated the material with a couple of new research and findings, some of them are as follows:

CPU Wait (new metric in 11g Performance page)
Latency issues on virtualized environments

So if you are attending the RMOUG training days, stop by at my session @ room 401 Thursday 9:45 am-10:45 am

There will also be a RAC Attack at the exhibition area so that must be fun

and don’t forget to follow RMOUG’s twitter @RMOUG_ORG for updates during the conference!

See yah!

Update:

I had a great time at RMOUG, was able to meet old/new friends and had a good number of attendees. The organizers really did a good job Here’s the updated PPT I used during the presentation.
That’s the most recent, the one on the RMOUG is not yet updated. For the scripts http://karlarao.wordpress.com/scripts-resources/ they are all at the “PerformanceAndTroubleshooting” folder

Thanks to all who attended my presentation, I know that’s a bit too much technical stuff for 1hour presentation squeezing all 101 slides and all that info. The best way is to go through my notes on the slides, check out all the links I pointed, and try to poke around on the scripts. And I assure you will learn tons of stuff & greatly appreciate the material :)

RMOUG – Mining the AWR

View more PowerPoint from karlarao

↧

Fast Analytics of AWR Top Events

March 24, 2012, 1:27 am

≫ Next: IOsaturationtoolkit-v2 with Exadata IORM and AWESOME text graph

≪ Previous: RMOUG 2012 training days

I’ve been working on a lot of good schtuff lately on the area of capacity planning. And I’ve greatly improved my time to generate workload characterization visualization and analysis using my AWR scripts which I enhanced to fit on the analytics tool that I’ve been using.. and that is Tableau.

So I’ve got a couple of performance and capacity planning use case scenarios which I will blog in parts in the next few days or weeks. But before that I need to familiarize you on how I mine this valuable AWR performance data.

Let’s get started with the AWR top events, the same top events that you see in your AWR reports but presented in a time series manner across SNAP_IDs…

So I’ve got this script called awr_topevents_v2.sql http://goo.gl/TufUj which I added a section to compute for “CPU wait” (new metric in 11g) to include the “unaccounted for DB Time” on high run queue or CPU intensive workloads. This is a newer version of the script that I used here http://karlarao.wordpress.com/2010/07/25/graphing-the-aas-with-perfsheet-a-la-enterprise-manager/

On the screenshots below I spooled the output on a text file and transformed it to a CSV format. Then from there you can play around with the data points using Tableau that you can download here http://www.tableausoftware.com/public/download

Is this a free software? Yes. But the catch is Tableau Public only allows you to have 100K rows on a given text or csv file. But for my capacity planning and troubleshooting purposes that’s enough to do the work The paid version of Tableau (Desktop) allows you to connect directly to the database but that will cost you $1999. ouch. The Tableau (Desktop Personal) is much like the Public version but with no data limit and it allows you to save your work on a specific file format (.twb and .tde files), Yes guys in Public version you are only allowed to save it to their Public portal… but that’s fine as long as I have the data points I can quickly work on the visualization again. Here’s the comparison if you are interested on the detailed product comparison http://www.tableausoftware.com/products/desktop/specs

Once you get the CSV file ready which you can download HERE

Open the Tableau and follow the steps below…

1) This is a trial version of Tableau Desktop.. the steps should be the same or similar with the Public version. Click “Connect to data”

2) Select the CSV file

3) Select “Text File”

4) Then you can view the data points

5) Data Points are shown below

6) Click OK

7) Import all data

8) It will create a file format that I believe where it stores the dimension data

9) After the data points import

10) Drag the “Events” on the color section, then “Add all members”

11) Click OK

12) Drag the AAS on the Rows

13) SNAP_ID on the columns. That’s your AWR SNAP_IDs

14) Filter the negative values. Those are the periods where you had instance shutdown.

15) Exclude the Nulls

16) And now you can select an area! hover on it and it will give you the event and the AAS number (the higher the AAS the more database work is happening)

17) Play around with the AAS data points

18) Now here it will show you detailed SNAP_ID data points (the dots) and you can hover on it

19) You can also view the underlying data points of those dots

20) Here it shows the event, SNAP_IDs, and the AAS value.. so here you can grab the SNAP_ID and execute the AWRRPT.sql on the database server to generate the larger AWR report

21) Drilling down more on the data points

22) You can also focus just on specific events. Let’s say just the CPU..

23) or the event that’s hogging most of the resources

24) and just select the data points from that bottleneck

25) Copy the summary info

26) Then play around with the data in command line

And clearly the database bottleneck is the PX event… which could just be the surface of the problem or a symptom for another issue..

And then visually you’ll start finding trends on the workload..

let’s say on this workload I found sustained high workload periods that would last for days..and these could be the periods where the batch jobs are executed..

SNAP_ID 34241 – 34306
11/07/06 03:30 – 11/07/07 12:00SNAP_ID 34417 – 34516
11/07/10 13:30 – 11/07/12 15:00SNAP_ID 34565 – 34694
11/07/13 15:30 – 11/07/16 08:00SNAP_ID 35213 – 35322
11/07/27 03:30 – 11/07/29 10:00

SNAP_ID 35363 – 35477
11/07/30 06:30 – 11/08/01 15:30

SNAP_ID 35590 – 35670
11/08/04 00:00 – 11/08/05 16:00

And if you want to have regression analysis, you must put all of these data points on the Y and X axis (I’ll create a script for that one of these days).. that way the r2toolkit will have a very good scope of data.. and then you’ll get your nice R2 trend..

but it doesn’t stop there..

you have to drill down on the data points, use the awrrpt, addmrpt, awrsqrpt, ashrpt, and drill down on sessions with snapper, os utilities, etc. to get to the bottom of the issue

so that’s forecasting+troubleshooting!

Update:

The screenshots above just makes sense of the SNAP_ID for the time series data. After a little more AWR data hacking I was able to make sense of the date value which is really useful because I no longer have think of SNAP_IDs across instances and getting them aligned on a particular period… and the trick here is Tableau automatically creates a time dimension for the time column of my AWR scripts. Kewlness

and that way I can stitch performance data across hostnames and consolidated instances of an entire Exadata Full Rack! and I’ll show how it is done on the next couple of posts.

The graph below shows an example of having a time dimension. Below is the AWR data (shown by day and hour) coming from a non-RAC database on a SAN storage with a default AWR retention period. The arrows below are pointing to a workload period where they are running a report which has a SQL that has access paths doing lots of multiblock IOs (db file scattered read) and that throttles the performance of the (already slow) storage causing the IO latency to shoot up. This graph was pretty useful on the storage performance troubleshooting which correlates to the numbers of OS (iostat, vmstat, and atop) and session level (snapper) tools. Now the single block latency (db file sequential read) has always been on the 20ms below range regardless of load. So fixing the SQL to favor the use of indexes (db file sequential read) made it do efficient IO, that made the SQL run from 15mins to 10secs.

Hope I’ve shared you some good stuff!

↧

IOsaturationtoolkit-v2 with Exadata IORM and AWESOME text graph

May 13, 2012, 10:04 pm

≫ Next: The effect of ASM redundancy/parity on read/write IOPS – SLOB test case! for Exadata and non-Exa environments

≪ Previous: Fast Analytics of AWR Top Events

I’ve got a new version of IOsaturation toolkit which you can download here http://karlarao.wordpress.com/scripts-resources/ and it has a cool script called “smartscanloop” that shows you the Smart Scan MB/s per database across the Exadata compute nodes.. it’s a per 2secs sample so that’s a pretty fine grained perf data and near real time text graph. Very useful for doing IORM demos and monitoring what database is currently hogging the IO resources and since it’s presented in a consolidated view you don’t have to go to each Enterprise Manager performance page and have a bunch of browser windows open.

The SECTION 1 is what I usually use to validate the IO numbers on the database side from my Orion (see oriontoolkit here) and Calibrate IO runs. I’ve been using it for quite a while on new RAC/non-RAC installations from client sites.. and I used it heavily on my R&D server while continuously enhancing the toolkit

The SECTION 2 gives you a standard tool to demonstrate the behavior of IORM (http://karlarao.tiddlyspot.com/#IORM).. so let’s say you are playing around with IORM percentage allocations for let’s say 3 databases the “saturate” script works well to generate load for each database and then you can observe the effects of the percentage allocation to the IO bandwidth/latency of each database.

when you run

./saturate 4 dbm1 2 exadb1

it will create 4 sessions on dbm1 and 2 sessions on exadb1 all doing parallel select and it outputs a log file for each database session. Each session log file has details on the start and end time, elapsed, MB/s which is pretty much everything you need to know to quantify the performance from a session level perspective. You’ll appreciate this session level output and be impressed on what IORM can do when you start investigating on IO prioritization as you see sessions from the other database having higher MB/s and lower elapsed times and as you play with different IORM scenarios.

cat *log | grep benchmark
Sample output below:
 benchmark ,instance       ,start            ,end              ,elapsed   ,MBs
 ----------,---------------,-----------------,-----------------,----------,-------
 benchmark ,dbm1           ,05/13/12 19:18:28,05/13/12 19:19:32,        64,    537
 benchmark ,dbm1           ,05/13/12 19:18:28,05/13/12 19:19:30,        62,    554
 benchmark ,dbm1           ,05/13/12 19:18:28,05/13/12 19:19:32,        63,    545
 benchmark ,dbm1           ,05/13/12 19:18:28,05/13/12 19:19:32,        64,    537
 benchmark ,exadb1         ,05/13/12 19:18:28,05/13/12 19:19:32,        64,    539
 benchmark ,exadb1         ,05/13/12 19:18:28,05/13/12 19:19:32,        64,    539

So the output of smartscanloop is the high level IO numbers across the cluster and the log files are your session detail numbers. Below is the simple output which just shows the SmartScan MB/s per database

This AWESOME text graph is similar to what you see in the Enterprise Manager performance page IO tab. Note that you’ll be seeing higher numbers of MB/s on the smartscanloop compared to EM because of a more fine grained interval (2secs) which is also the same behavior when you measure the IO latency as I explained here (avg latency issue)

then I’ve modified the script to have the advanced output that shows the Hierarchy of Exadata IO. See the updated README for more details on how to use it. Below is the output

What’s good about this is the numbers are about the same when you do a per 10secs snapshot of AWR.. compare the AAS and latency (avgwt ms) columns of the above image and below

Again, this is pretty useful for monitoring the high level smart scans IO that’s happening across your Exadata cluster, if you are on an environment where there’s separation of duties you can even hand off this script to the sys admins that are monitoring the storage cells with their home grown alerting scripts, kSar, or nagios.. so this will serve as their view on the database side of things.

And if any of your clients haven’t adopted the IORM, this is very useful for DEMOs to customer sites to showcase the IORM capabilities.. and if you don’t want to show the latency and other columns you can opt to just use the simple output which only shows the smart scans MB/s (see get_smartscan.simple on README ). Most of the time.. the simpler the output the easier for them (users) to understand.

Sweet! right?!?

Wait.. Does the toolkit work on non-Exadata DBs?

The SECTION 1 works on Exa and non-Exa (I’ve also mentioned this at the beginning of this post)

The SECTION 2 works well on Exa because the underlying scripts just makes use of dcli commands, some shell and SQL.. but if you are interested to have the smartscanloop output on non-Exa environment I have the script get_smartscan.nonexa on the toolkit that shows the “physical read total bytes” instead of the “cell physical IO bytes eligible for predicate offload”..

then on each node do this if it’s a non-Exa RAC

$ while : ; do ./get_smartscan.nonexa | grep "%" ; echo "--" ; sleep 2 ; done
%,05/12/12 12:57:33,DEMO1     ,      0,
%,05/12/12 12:57:35,dbm1      ,      0,
%,05/12/12 12:57:38,DBFS1     ,      0,
%,05/12/12 12:57:40,exadb1    ,      0,
%,05/12/12 12:57:42,PRIMARY1  ,      0,
--
%,05/12/12 12:57:46,DEMO1     ,      1,
%,05/12/12 12:57:48,dbm1      ,   2078, @@@@@@@@@@@@@@@@@@@@@
%,05/12/12 12:57:50,DBFS1     ,      0,
%,05/12/12 12:57:52,exadb1    ,    972, @@@@@@@@@@
%,05/12/12 12:57:55,PRIMARY1  ,      0,
--
%,05/12/12 12:57:59,DEMO1     ,      0,
%,05/12/12 12:58:01,dbm1      ,   1088, @@@@@@@@@@@
%,05/12/12 12:58:03,DBFS1     ,      0,
%,05/12/12 12:58:05,exadb1    ,   2356, @@@@@@@@@@@@@@@@@@@@@@@@
%,05/12/12 12:58:07,PRIMARY1  ,      0,
--
%,05/12/12 12:58:11,DEMO1     ,      0,
%,05/12/12 12:58:14,dbm1      ,   1000, @@@@@@@@@@
%,05/12/12 12:58:16,DBFS1     ,      0,
%,05/12/12 12:58:18,exadb1    ,   2336, @@@@@@@@@@@@@@@@@@@@@@@
%,05/12/12 12:58:20,PRIMARY1  ,      0,
--
%,05/12/12 12:58:24,DEMO1     ,      0,
%,05/12/12 12:58:26,dbm1      ,   1031, @@@@@@@@@@
%,05/12/12 12:58:29,DBFS1     ,      0,
%,05/12/12 12:58:31,exadb1    ,   2071, @@@@@@@@@@@@@@@@@@@@@
%,05/12/12 12:58:33,PRIMARY1  ,      0,
--

But wait! there’s more!

Since the output is in comma separated format.. you can just do the following

./smartscanloop > smartscanloop.txt

and leave it running… So you can do this when you want to characterize the IO of a batch run or reporting of multiple databases and the text file that will be generated is pretty small and it doesn’t hurt in terms of CPU/IO resources because it’s just doing distributed SSH and doing some SELECT on v$ views

then from the text file you can easily see what’s the highest MB/s by executing the command below


$ less smartscanloop.txt | sort -nk5 | tail
pd01db03: %,05/10/12 21:29:28,biprd2    ,   7358, @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
pd01db03: %,05/11/12 08:57:12,biprd2    ,   7363, @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
pd01db03: %,05/10/12 14:55:54,biprd2    ,   7454, @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
pd01db03: %,05/11/12 09:29:57,biprd2    ,   7556, @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
pd01db03: %,05/10/12 18:17:50,biprd2    ,   7785, @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
pd01db03: %,05/10/12 19:12:19,biprd2    ,   7880, @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
pd01db03: %,05/11/12 10:31:24,biprd2    ,   7886, @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
pd01db03: %,05/10/12 19:15:48,biprd2    ,   8112, @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
pd01db03: %,05/11/12 10:31:40,biprd2    ,   8138, @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
pd01db03: %,05/10/12 18:49:46,biprd2    ,   9315, @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

and since it’s a CSV file you can then easily graph it in Tableau! (just put a header on it first)


host       , time            , inst     , smartscan,
pd01db01: %,05/10/12 13:08:14,hcmprd1   ,      0,
pd01db01: %,05/10/12 13:08:16,paprd1    ,      0,
pd01db01: %,05/10/12 13:08:18,rmprd1    ,      0,
pd01db01: %,05/10/12 13:08:20,lmprd1    ,      0,
pd01db01: %,05/10/12 13:08:22,DBFSPRD1  ,      0,
pd01db01: %,05/10/12 13:08:24,hcrprd1   ,      0,
pd01db02: %,05/10/12 13:08:14,hcrprd2   ,      0,
pd01db02: %,05/10/12 13:08:16,rmprd2    ,      0,
pd01db02: %,05/10/12 13:08:18,paprd2    ,      0,
pd01db02: %,05/10/12 13:08:20,hcmprd2   ,      0,
pd01db02: %,05/10/12 13:08:22,DBFSPRD2  ,      0,
pd01db02: %,05/10/12 13:08:24,lmprd2    ,      0,
pd01db03: %,05/10/12 13:08:14,biprd2    ,   3412, @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
pd01db03: %,05/10/12 13:08:16,DBFSPRD3  ,      0,
pd01db04: %,05/10/12 13:08:14,DBFSPRD4  ,      0,
pd01db04: %,05/10/12 13:08:16,mvwprd2   ,      0,
pd01db04: %,05/10/12 13:08:18,fsprd2    ,    763, @@@@@@@@
pd01db04: %,05/10/12 13:08:20,mtaprd112 ,      0,
... output snipped ...

and hack the data! remember on the advanced output you can slice and dice it by host, time, inst, aas, latency, returned, ic, smartscan..

Here’s the IO graph per instance

and IO graph per instance and host

Hope I’ve shared you some good stuff

↧

The effect of ASM redundancy/parity on read/write IOPS – SLOB test case! for Exadata and non-Exa environments

June 28, 2012, 9:22 pm

≫ Next: Speaking at E4!

≪ Previous: IOsaturationtoolkit-v2 with Exadata IORM and AWESOME text graph

Last week I had a lengthy post at oracle-l that tackles Calibrate IO, Short Stroking, Stripe size, UEK kernel, and ASM redundancy effect on IOPS Exadata which you can read here
followed by interesting exchange of tweets with Kevin Closson here (see 06/21-22 tweets) which I was replying in between games at UnderwaterHockey US Nationals 2012 which we won the championship for the B division I have my awesome photo with the medal here

This post will detail on the ASM redundancy/parity effect on IOPS… if… by changing the ASM redundancy (external, normal, and high) will it decrease the workload read/write IOPS or stay as is. I’ll walk you through the step by step on how I did the instrumentation + the test case itself then end with some results and observations.

Let’s get started!

The Environment

My R&D server running OEL 5.7 on UEK kernel with

Intel 2nd gen Sandy Bridge
8 x 1TB 7200 RPM SATA3 – shortstroked
- on first 320GB vbox LVM stripe (VirtualBox .vdi files)
- next 320GB ASM (DATA disk group – external redundancy)
- remaining RECO area (for backups)
16GB physical memory having lots of VirtualBox VMs (12 VMs can run concurrently with 0.45 load average and no CPU WAIT IO)

SLOB installation

Follow the “Installation” and “Run the benchmark” sections on my wiki here http://karlarao.tiddlyspot.com/#%5B%5Bcpu%20-%20SillyLittleBenchmark%20-%20SLOB%5D%5D

Instrumentation

Essentially you have to be able to instrument the different layers of things. The sample screenshot below are the terminal sessions I have while doing the test cases and it’s really helpful to have them all in a single view because you’ll be able to quickly notice changes as the workload grow and reach its sustained state. So when you run sh ./runit.sh 0 128 you’ll be able to see from load average of 2 it will gradually go up to 128 and it will stay on that range and that’s the sustained state of the workload. Then from there I’ll start to take screenshots and gather all sorts of data that will serve as my data points from that test case. So with the single view you’ll be able to see how everything behaves from the ramp-up to the sustained state and when you cancel the run or whenever you do any changes to your test case environment.

1) Hardware:

On the hardware side I’ve got collectl (http://collectl.sourceforge.net) which shows the overall workload of the OS and CPU + IO details in a time series manner (see the first 3 terminal windows above). And I usually execute the following commands for every run and package the spooled output in a test case folder

collectl --all -o T -o D >> collectl-all.txt
collectl -sc --verbose -o T -o D >> collectl-cpuverbose.txt
collectl -sD --verbose -o T -o D >> collectl-ioverbose.txt

The critical numbers would be the aggregate IOPS from all the 8 disks, and the detailed IOPS per disk together with the IO size, service time, and Pct% Utilization. All these numbers gives you an IO characterization of what SLOB is doing.

2) Database:

Load Profile
The OS numbers has to be correlated to the physical reads/writes (load profile) on the database side and just relying on the AWR snapshots taken at start and end of the test case is not really a good measure especially on a small scale box like mine with just 4cores and 8spindles which essentially I’m making it work for 128 concurrent users doing an update on their own tables.. and ultimately the queueing effect will kick in and the disks will be really busy that you’ll get to a point where the AWR snapshots will not be able to do its thing and may even hang or queue or it will wait for it to be able to grab some IO or until it gets an IO bandwidth to do its own operations then that’s the time it will be able to complete the AWR snapshot/report…whew.

So to get around that, you have to read those performance numbers from memory and it’s only possible by doing a query from a v$ view.. luckily there’s a script created by Timur Akhmadeev a few months ago http://timurakhmadeev.wordpress.com/2012/02/21/load-profile/ that outputs the same Load Profile section of the AWR report but pulls data from the v$sysmetric view. The view outputs the past 15 and 60 seconds of a bunch of metrics which is similar to V$SYSSTAT but some metrics are already transformed into per/sec or per/transaction numbers. The one that Timur used is the 60secs time slice which is equivalent to creating 1min AWR reports which is good enough since the workload is sustained and the numbers will most likely not vary that much plus the Load Profile section has all the stuff that I need to see what’s happening on the physical reads/writes because SLOB is essentially a test case to guage your reads/writes IOPS.

I’ve modified the SQL https://www.dropbox.com/s/3ijoy3zljqps0wx/loadprof.sql to add the “tm” column so I can correlate it with the time series output of collectl. The sample output is shown on the left side of the image above. You can download the script and put it on the SLOB directory and run it as

while : ; do sqlplus "/ as sysdba" @loadprof ; echo "--"; sleep 2 ; done

ASH
I also want to see how the workload looks like in ASH.. so I’ve got a script called ash that calls a SQL script by Kyle Hailey which gives you the Average Active Sessions (AAS) for the past 5 seconds time slice.. so if let’s say I’m doing a sustained number of 128 readers I should be able to see 128 AAS doing “db file sequential read” (random IO).. so rather than pulling up an OEM or creating a repo for OEM just to see what’s happening on the SLOB run I can just go with this text graph or text version of what I’d like to see..

The sample output is shown on the right hand side of the image, see the FASL column having the 128 AAS on “db file sequential read”.. also the time column at the far left can be correlated to the collectl and loadprof.sql output. The script is available here
https://www.dropbox.com/s/pbcewv374dq5bc0/ash-aveactn300.sql

3) Sessions:

Another important thing is measuring the latency… I could have written a SQL that will mimic the top 5 timed events pulling from v$ views that also shows the latency column but for now I don’t really have the time to do that. So a very good alternative is to do it with the “awesome snapper” then spool it to a file and grep for stuff… and because the SLOB is doing a sustained reads/writes workload the numbers will most likely stay the same or within the range of that sustained number and with that I can just do 5 samples or 5 runs of snapper and that will tell me what my latency is.

Good thing about using the snapper is the latency is being measured from the SLOB users themselves and that’s what the sessions are experiencing for each IO operation that they are doing. Also take note that with the snapper command below I can grep for DBWR, CKPT, LGWR processes which I can’t do in the usual top 5 timed events..which I was able to build a cool comparison of latency graphs across redundancy levels (more on this on the results section). And each snapper sample has a timestamp that you can correlate with the collectl, loadprof, and ash output. Below are the commands I used:

spool snapper.txt
@snapper ash=event+wait_class,stats,gather=tsw,tinclude=CPU,sinclude=redo|reads|writes 5 5 "select sid from v$session where username like 'USER%' or program like '%DBW%' or program like '%CKP%' or program like '%LGW%'"
spool off

cat snapper.txt | grep write | egrep "DBW|CKP|LGW" | grep WAIT | sort -rnk2
cat snapper.txt | grep "db file sequential read"
cat snapper.txt | grep "free buffer waits"
cat snapper.txt | grep "write complete waits"

Check out the script here http://tech.e2sn.com/oracle-scripts-and-tools/session-snapper

The test case

Before the test case I executed a backup of the SLOB database then I make use of that backup as a starting point of each iteration which makes the underlying data consistent across runs. Then for each ASM redundancy I did the following:

1) Wipe out the diskgroup

sqlplus "/ as sysasm"
alter diskgroup data dismount force;
drop diskgroup data force including contents;

2) Create the new diskgroup (1st iteration – normal, 2nd iteration – high, 3rd iteration – external)
3) Restore the backup
4) Execute 128 readers and gather perf data
5) Execute 128 writers and gather perf data
6) Go back to step1

Results

>>128 Readers Testcase

External Redundancy

Normal Redundancy

High Redundancy

>>128 Writers Testcase

External Redundancy

Normal Redundancy

High Redundancy

>>Read IOPS

Shown on the left is the IOPS comparison across redundancy
on the right is the read:write ratio

>>Write IOPS

Shown on the left is the IOPS comparison across redundancy
on the right is the read:write ratio
the graph at the bottom is just the “write iops” part of the graph above.. since SLOB is executing a massive UPDATE SQL with some filter predicates and that causes scan on some rows which translates to “read iops”

>>Read Latency (ms)

Here are the graph definitions:
- ckpt -> CKPT process – “control file parallel write” latency
- lgwr -> LGWR process – “log file parallel write” latency
- dbwr -> DBWR process – “db file parallel write” latency
- phyreads -> SLOB users – “db file sequential read” latency
- fbw -> SLOB users – “free buffer waits” latency
- wcw -> SLOB users – “write complete waits” latency

>>Write Latency (ms)

(see graph definitions above)

Observations

>>Some insights on the numbers above…

First of all.. I can say I did a really good job on documenting all these stuff the results are in 6 parts

part 1 >>128 Readers Testcase
part 2 >>128 Writers Testcase
part 3 >>Read IOPS
part 4 >>Write IOPS
part 5 >>Read Latency (ms)
part 6 >>Write Latency (ms)

but as you can see at a high level as I change redundancy the read IOPS stayed at the range of 2400+ IOPS while on the writes as I move to “normal” redundancy it went down to half and “high” redundancy it went down to 1/3 and if you are a big SLOB fan, the first two parts of the results will be gold mine for you because that’s more of a geeky output and I essentially pulled the numbers from those raw screenshots to build the graphs.

So let’s move ahead on explaining the charts…

On the Read IOPS and latency I don’t see any huge difference and I can say it’s not being affected.. the IOPS stayed at the range of 2400+ IOPS then the latency at the range of 900ms across redundancy levels. It clearly shows that you don’t have a raid penalty or parity penalty to sacrifice for reads.

On the Write IOPS and latency

it’s pretty clear that on normal you lose half and on high you only get 1/3 of the hardware IOPS
And what’s interesting about this is when I was watching the workload of normal and high I can see an increasing number of AAS (average active sessions) spent on “free buffer waits” and “write complete waits”… well I’m seeing them on external redundancy but not as much.
Actually, as we change redundancy we are lowering the IOPS capacity which we sacrifice for the redundancy/parity and still streaming the same amount of workload and these wait events are just signs of inadequate bandwidth. The DBWR is working harder to make free buffers available for the same workload with lesser IOPS to work on and that translates to higher latencies.
You will also notice on the raw screenshots that the hardware write IOPS numbers from collectl stayed at around 1000+ at full throttle while on the database load profile you’ll see that it’s just around 300 (high redundancy) which is the effect of the ASM parity.

>>How can this affect your Storage Capacity Planning?

I’ll pull a conversation I had with a really good friend of mine.. his question was ”Quick question on your AWR mining script AWR-gen-wl…, is IOPs calculated before or after ASM mirroring? For example on Exadata, if I see 10,000 write IOPs, did the cells do 10k or did they do 20k (normal redundancy)?”

and here’s my response..

The script awr_iowl.sql and awr_iowlexa.sql have the columns that accounts for RAID1.. that is read penalty of 1 and write penalty of 2.
read on the section “the iops raid penalty” on this link http://www.zdnetasia.com/calculate-iops-in-a-storage-array-62061792.htm and the “real life examples” on this link http://www.yellow-bricks.com/2009/12/23/iops/

so those computations should also apply for Exadata since normal redundancy is essentially RAID1 that’s write penalty of 2, and the high redundancy is penalty of 3.

Now I remember this sizing exercise I had with an EMC engineer on a project bid before https://www.evernote.com/shard/s48/sh/03602b99-3274-4c64-b5d1-bbe7bd961f8d/95be02ccf9aa75cf863bb19115353eb0
and that’s why I created those columns to get the data directly from AWR.. so for every snapshot you’ve got the “hardware iops needed” and “number of disks needed”, what’s good about that is as your workload vary those two numbers will be representative of that workload. So since you have a lot of data samples, I usually make a histogram on those two columns and get the top percentile numbers because most likely those are the peak pereiods and I can investigate on it by drilling down on the snap_ids and looking into the SQLs and validating it to the app owners as to what’s the application is running at that time.

I’ve attached an excel sheet which you can just plug the total workload iops on the yellow box. So in your case, let’s say you have 10K workload IOPS with 50:50 read/write ratio… that’s equivalent to 15K hardware IOPS for normal redundancy and 20K hardware IOPS for high redundancy.

the excel screenshot is actually here —-> https://lh6.googleusercontent.com/-00PkzwfwnOE/T-N0oo2Q-FI/AAAAAAAABqE/EbTOnHBlpmQ/s2048/20120621_IOPS.png

Note that I’m particular with the words “workload IOPS” and “hardware IOPS”
so on this statement
“if I see 10,000 write IOPs, did the cells do 10k or did they do 20k (normal redundancy)?” <– if this 10,000 is what you pulled from the AWR then it’s the database that did the 10K IOPS so that’s the “workload IOPS”.. and that’s essentially your “IO workload requirements”.

Then let’s say you haven’t migrated to Exadata.. you have to take into account the penalty computation shown above.. so you’ll arrive with 15000 “hardware IOPS” needed (normal redundancy).. and say each disk has IOPS of 180 then you need at least 83 disks so that’s 83disks / 12 disks each cell = 6.9 storage cells … and that’s Half Rack Exadata. But looking at the data sheet https://www.dropbox.com/s/ltvr7caysvfmvkr/dbmachine-x2-2-datasheet-175280.pdf it seems like you can fit the 15000 on a quarter rack (because of the flash).. mmm.. well I’m not pretty confident with that because if let’s say 50% of 15000 IOPS are writes (7.5K IOPS) then I would investigate on the IOPS write mix if most of it are DBWR related (v$sysstat.physical write IO requests) or LGWR related (v$sysstat.redo writes) and if most of it are DBWR related then I don’t think you’ll ever benefit from the smart flash log. So I would still go with the Half (12500 disk IOPS) or Full Rack (25000 disk IOPS) for my “hardware IOPS” capacity. And I’ll also take into consideration the MB/s needed for that database but that should be augmented by the flash cache.

In addition to my response above.. Also be aware that Smart Flash Log does not lower the HDD (HardDisk) touch load…both sides of the write go to flash and HDD concurrently. So you should not be too optimistic about the Smart Flash Log when sizing Exadata for “write IOPS” because you can’t predict who will always win on the fastest-write-race…so as much as possible base your write IOPS sizing on HardDisk IOPS. But definitely on the read IOPS even small (OLTP) or large IOPS (DSS) the flash cache will be able to augments those HDD numbers.

So with my test case shown here.. the statement on the reply above is true. The ASM redundancy/parity affects the “workload write IOPS” number (be it small-OLTP or large-DSS IOPS).. but not for the “workload read IOPS”. This behavior is the same even on a regular SAN environment… which you have to be careful/aware when sizing storage. So now you know that 50K IOPS in any data sheet is not equal to 50K write IOPS.. factor in the redundancy and always remember Utilization=Requirements/Capacity no matter what the type workload or platform is as long as as the total workload IOPS/MBs requirements is well within the total hardware IOPS/MBs capacity you are up a notch on not seeing any IO bottlenecks.

Hope I’ve shared you some good stuff

↧

Speaking at E4!

June 29, 2012, 1:42 pm

≫ Next: OakTable World 2012

≪ Previous: The effect of ASM redundancy/parity on read/write IOPS – SLOB test case! for Exadata and non-Exa environments

I’ll be co-speaking with Randy Johnson (one of the authors of Expert Oracle Exadata) at E4 to share about the war stories and detail on a bunch of technical stuff on a Peoplesoft and BIEE consolidation project we had on one of our clients. See the abstract below:

Randy Johnson & Karl Arao
A PeopleSoft & OBIEE Consolidation Success Story
In today’s competitive business climate companies are under constant pressure to reduce costs without sacrificing quality. Many companies see database and server consolidation as the key to meeting this goal. Since its introduction, Exadata has become the obvious choice for database and server consolidation projects. It is the next step in the evolutionary process. But managing highly consolidated environments is difficult, especially for mixed workload environments. If not done properly the quality of service suffers. In this session we tell the tale of a large real estate investment company that successfully consolidated their global operations onto a Maximum Availability Architecture Exadata platform. Applications sharing this environment include PeopleSoft Financials, PeopleSoft HR, Portal, and OBIEE. Accurate provisioning and management of system resources was absolutely essential to our success. In this session we share lessons learned and the tools you’ll need to ensure that your consolidation story has a happy ending.

E4 is an Exadata focused event and we have a very interesting speaker line up and abstract list http://www.extremeexadata.com/e4/abstracts which are powerful & useful for Oracle DBAs/Architects/Managers who are seriously into performance and Exadata not just from a “wow that’s interesting” but also from a practical perspective. So I’m pretty sure you Exadata hackers out there will surely love this event because it’s going to be all technical!

Just to give you a glimpse on what Randy and I are presenting, I have here a mind map of our topic list which we will transform into an interesting story..

I’ll be blogging some of them in the next couple of days.. the ASM writes IOPS effect is already out and I’d like to focus next on the CPU capacity planning… taking the quantitative information and make it visual so both the technical and non-technical can “see” what’s going on leading to solid solutions.

See you at E4!

↧

OakTable World 2012

September 27, 2012, 2:35 pm

≫ Next: Oracle Big Data Appliance First Boot

≪ Previous: Speaking at E4!

This Monday Oct 1, 2:00-2:50PM I’ll be presenting at the OakTable World which used to be called as Oracle Closed World, this year it’s a full two-day conference with awesome speakers lineup and very interesting topics… you can just watch this video by Mogens Norgaard to know more about the roots of this very technical event.

My presentation title is “Where did my CPU go?” – monitoring & capacity planning adventures on a consolidated environment .. and If you love to hack the ASH and AWR data and curious about CPU capacity planning then you’re really going to enjoy this session.. it’s going to be all about CPU

For the general flow of my presentation, I’ll be discussing about the ASH math as it relates to the CPU from the Oracle and OS perspective. I’ll explain the theory (see the doodle below) and show/demo how this works with real numbers/test case (take note of the 30secs sessions below) on a real hardware (my R&D server).

then once you get the ASH math and CPU foundation I’ll introduce the important CPU metrics from the Oracle perspective and how it relates to capacity planning

and from here I’ll show some CPU capacity planning scenarios using those important metrics and tools that can help slice and dice the performance data. Much like a self service database analytics

I’m pretty sure this will be a fun hacking session.. and it gets even better because there will be tons of freebies (tshirt, JL’s “Oracle Core” book, DB Optimizer software, beer and food) on this FREE event

C’mon & Register HERE: OakTable World

See ya!

↧

Oracle Big Data Appliance First Boot

October 16, 2012, 4:38 am

≫ Next: RMOUG, IOUG Collaborate, KSCOPE, and E4 2013

≪ Previous: OakTable World 2012

This is a quick video of our Big Data Appliance first boot yesterday in our server room (with Kerry and Andy), now she’s beside our two Exadata quarter racks all we have to do now is connect both of them through the infiniband spine switch located at the bottom of each of the rack. At the latter part of the video you’ll hear me counting as I power on the top half of the rack, I’ve seen enough blinking green lights yesterday and that’s what I call “server porn”.. thinking about how can I push the limits of those servers and all the fancy monitoring and workload characterization I can do with it.. but of course this is everybody’s R&D machine at Enkitec Andy and I had a quick look on the server, it comes bundled with a particular image version and running on UEK kernel (OEL 5.6).. the machines are not yet configured, and it has a similar Exadata onecommand utility to configure the entire stack which the doc names it as the “Mammoth Utility” http://docs.oracle.com/cd/E36174_01/doc.11/e36161/mammoth.htm#BABFCEGJ which will install and configure the Cloudera Distribution including Apache Hadoop plus the other Big Data stuff (connectors, NoSQL database, etc.)

Now if you are curious on what this machine can do.. check out this white paper -> 12TB/hour data load High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database http://www.oracle.com/technetwork/bdc/hadoop-loader/connectors-hdfs-wp-1674035.pdf

That’s something we can replicate on our lab environment. This will be fun!

↧

RMOUG, IOUG Collaborate, KSCOPE, and E4 2013

February 4, 2013, 10:29 pm

≫ Next: Speaking at E4 2013! … and some Exadata Patents good stuff

≪ Previous: Oracle Big Data Appliance First Boot

I’ve been really busy lately doing a lot of stuff, and most of the time if I’m doing some intensive research on a topic I would just put them all in my braindump (wiki) so if you’ve been following this blog might as well subscribe on my wiki RSS because there are just a bunch of bits of research going-on (see this mindmap) and what usually happens is I put them all together to make an awesome presentation/white paper.

So this 2013.. I’ll be presenting at RMOUG, IOUG Collaborate, and KSCOPE 2013

If you’ll be at RMOUG next week make sure you have the Guidebook app on your phone then search for “RMOUG Training Days 2013″ and clicking on it will download the whole guide which has all the details you need to make the most out of a very technical conference the app is awesssomeee, and you can take a pic, tweet, maps, facebook, and most of all my schedule is lined up in one page and very searchable. My schedule is below:

After the lunch with the ACEs on Tuesday, I’ll be hanging out with Tim Gorman, Jeff Smith, and Kris Rice to talk about database careers and I believe for the past few years we’ve been having High School students attend the whole conference so having a career talk would be a good avenue for mentorship, motivation, and some career guidance. And we’ve got some interesting topics and talking points lined up for you guys so this will be really interesting

Then the next day (and I promise to not drink beer the night before) I’ll be delivering two presentations.

The first one is “Where did my CPU go?” which I’ll talk about how to compare between various CPU models and hardware platforms (both actual benchmarking and a reasonable method of quickly estimating relative performance based on SPECint_rate2006 and TPC-C results), then detail on the performance differences of cores vs threads, then move on to the database specific stuff which I’ll discuss the different CPU events, and lastly the monitoring/capacity planning of CPU on massively consolidated environments.
The 2nd is “A Consolidation Success Story” which covers some very interesting war stories about consolidating mixed workload databases in Exadata

Oh! and lastly, the Enkitec Extreme Exadata Expo will take place on August 5-6, 2013 in Four Seasons Hotel & Resorts in Irving, TX. And the call for papers is open, so if you have something interesting that you’d like to share, please do submit your abstract!

Below is the video of last year’s event, it’s a compilation of all the presenters and the abstract titles and you’ll see me and Randy Johnson @3:20

Hope I’ve shared you some good stuff

↧

Speaking at E4 2013! … and some Exadata Patents good stuff

May 23, 2013, 11:21 am

≫ Next: OOW and OakTable World 2013

≪ Previous: RMOUG, IOUG Collaborate, KSCOPE, and E4 2013

I’ll be co-speaking with Tyler Muth at E4 and we will be talking about Performance Visualization… some of you may already know that I’m really a big fan of Tableau and just to give you an example on what the tool can do I’ve consolidated all the different viz that I’ve done during my sizing & performance gigs here http://goo.gl/xZHHY and Tyler has also been doing a lot of Exadata performance & benchmarking gigs lately and there were a couple of times where we collaborated on specific viz (him doing it on R) just to validate each other’s work. In short, we are passionate about this stuff. And we are going to be awesome

The session title is: Performance data visualization with Oracle R and Tableau

and here’s the abstract:

In this session Tyler and Karl will do an example based and problem/solution approach in showing how to visualize performance data from Exadata benchmarking and some of their most interesting production workload characterization and sizing scenarios. Each problem/scenario will be visualized using Oracle R and Tableau and will drill down deep on how to effectively explore the data and match it with the different ideal visualization types leading to quantitative information and solid solutions.

Also here’s a little tidbit about some of the speakers.. yesterday I was compiling a list of people who invented the main technologies of Exadata and side by side with their patents. This started when I can’t find enough documentation about the write-back flash cache… even the Exadata official doc and the MOS FAQ 1500257.1 are not enough so I went on and search further and was able to find some gold mine. Here’s the list that I have so far:

Roger Macnicol http://goo.gl/oxxu7
hcc http://goo.gl/9ptFe, http://goo.gl/3IOSi

Sue Lee http://goo.gl/6WCFw, http://goo.gl/bI0pd
iorm http://goo.gl/BHIc1

Boris Erlikhman http://goo.gl/2LvXU
smart scan http://goo.gl/chy2s
flash cache http://goo.gl/YlCA7
smart flash log http://goo.gl/TwyRx
write back cache http://goo.gl/2WCmw

What’s interesting here is that the first two people on the list will be speaking at E4 talking about technical deep stuff now that’s just superb.

Of course Tom Kyte will be this year’s keynote speaker and especially Doug Cutting, the creator of Hadoop, will be joining the fantastic list of speakers.

The event takes place August 5-6, 2013 at Four Seasons Hotel, Irving, Texas and there is plenty of time for you to register.. that’s $850 Live / $499 Virtual but of course if you attend live then you’ll be able to drink beer with us which will get even more technical And you also have the option to attend the intense Training Days http://www.enkitec.com/e4/training-days to complete the week!

See ya!

↧

OOW and OakTable World 2013

September 22, 2013, 7:15 am

≫ Next: my Time/Task/Goal/Habit (TTGH) management

≪ Previous: Speaking at E4 2013! … and some Exadata Patents good stuff

Today is the 1st day of OOW for me, and there are a lot of cool stuff going on.

Sessions:
——————–

Later I’ll be doing an IOUG session at Moscone West talking about CPU on how to monitor and do capacity planning and sizing. This is a pretty interesting topic and I will give you enough foundation to be able to consolidate a bunch of databases on one server or a cluster. There’s a method to this and you can be pretty accurate about it. Remember 12c is geared towards consolidation but it doesn’t really change the fundamentals when monitoring and sizing CPUs.. I’ll just talk more about that later.

Session ID: UGF9790
Session Title: Where Did My CPU Go?
Venue / Room: Moscone West – 3014
Date and Time: 9/22/13, 15:30 – 16:30

And on Tuesday I’ll be presenting at OakTable World.. I’m very excited about this! I’ll do a session highlighting a very write intensive OLTP Exadata environment and will discuss the different ways to monitor IO from the database and storage layer perspective and correlating it back to the application by mining the dba_hist_sqlstat data. I’ll also touch on utilizing the OEM12c Metric Extensions and BI Publisher integration to ultimately scale the monitoring to a bunch of Exadata environments. It’s going to be a fun hacking session.

http://www.kylehailey.com/oaktable-world/agenda/

Other Activities:
——————–

In just a few minutes I’ll be doing the Open World Bridge Run https://www.facebook.com/events/202355076600435/ Sunday, September 22, 2013 Time8:00am in PDT

Later this evening is the ACE dinner Sunday at The Walt Disney Family Museum, September 22 from 6:15 PM to 9:30 PM

Tomorrow the Swim by the bay https://www.facebook.com/events/122846764577904/ Monday, September 23, 2013 Time7:30am in PDT

Also Kent Graziano’s Chi Gung either at Union Square or Yerba Buena Gardens every 7am

And if you have an Oraclenerd gear make sure to join us at https://www.facebook.com/events/175411045980422/

On the 25th is the Annual Bloggers Meetup @ Jillian’s http://www.pythian.com/blog/oracle-openworld-2013-bloggers-meetup/

And I’ll be hanging out at the RAC Attack to build some 12c RAC and meet the awesome Ninjas that made this happen https://www.facebook.com/racattackoow13

Enkitec @ OOW and OakTable World:
—————————————-

Enkitec is at Booth #501 at Moscone South where we will feature continuous Oracle Exadata and Oracle APEX demos. And you’ll also get to meet a bunch of Enkitec peeps hanging out so just drop by and we can talk about Exadata stuff!

We look forward to seeing you and make sure to check out the Enkitec activities/sessions here http://www.enkitec.com/about/news/enkitec_at_oracle_openworld_2013_san_francisco

Have a great OOW/OakTable World week!

↧

my Time/Task/Goal/Habit (TTGH) management

December 4, 2014, 3:15 pm

≫ Next: Oracle-Validated RPM on OEL 4.5

≪ Previous: OOW and OakTable World 2013

There was a post last night on oracle-l on the topic of task/project/time management, and I’m a big fan of productivity hacks and the whole GTD idea or method http://en.wikipedia.org/wiki/Getting_Things_Done. The reason for adopting this method is I’ve been on consulting business my entire career and I always get swamped with a lot of work from multiple customers or research stuff and it’s really difficult to prioritize and focus without a system. The same thing I learned how to speed read (see tony buzan book) because there’s just not enough time and you want to learn as much as you can. The bottom line is you want to be efficient.

So I thought of sharing my way of dealing with this in general…

below is my post on oracle-l

I would start with Tom Limoncelli’s Time Management for System Administrators, he’s got a video presentation here

http://youtu.be/XMc7jw38Bxs?t=2m48s that became the foundation on how I manage my tasks

Personally I have Goal, Habit, Task, Time Trackers

And there are two parts to this: having a system that works for you and being able to instrument that

1) you have to have a system on setting your goals and prioritizing your tasks, and this one is a great response from quora http://www.quora.com/Productivity/As-a-startup-CEO-what-is-your-favorite-productivity-hack/answer/Paul-A-Klipp?srid=n2Fg&share=1

now I’m using mindmaps for goal setting and kanbanflow.com for task management which I really like because you can specify swimlanes+colors which makes it kind multidimensional than just a regular calendar
before I just use google calendar for tasks and “goals on track” for goal/habit tracker but I unsubscribed from that service and migrated the entire workflow to mindmaps (I use freemind)

2) you have to have a way of instrumenting your time

on my windows VM where I do all my work I have ManicTime installed and this enables me to track everything I’m doing.. automatically without any user input and it can auto tag applications let’s say if I open putty the time I spent on that app will be tagged as “Work”, and I can see where my time went just by graphing the data https://www.evernote.com/l/ADBlN746vCxDXJykSPwZMT4TFUMQ6xT9oVw
on my mac host I have this free version of RescueTime, I like the weekly productivity percentage being emailed every week usually I’m about 68% per week.. If I go below, that means I’m pretty lazy that week. Above that means I was pretty busy
kanbanflow on the other hand forces you to input your tasks + the corresponding time you spent on it. So what I would do is at the end of the day I would export the data and graph it on my tableau dashboard. I just need to open the dashboard I created and it will automatically read the new file and it looks like this https://www.evernote.com/l/ADD5nUeDwrZLpoc87uhpsqdKeHeNvvMPJcI on that link you’ll see the entire workflow I have for task management up to visualization

Some of these may not work for you, but at least you’ll get the high level idea. So a couple of years back early on my career I was learning and coming up with my own systems and using these tools, through the years I’m also improving and hacking it and up until now that helps me getting things done (GTD) and motivated.

↧

Oracle-Validated RPM on OEL 4.5

November 7, 2008, 2:17 am

≫ Next: Migrate from Windows XP 64bit to Ubuntu Intrepid Ibex 8.10 64bit

≪ Previous: my Time/Task/Goal/Habit (TTGH) management

This is officially my first post on this blog.. finally after weeks of procrastination ..So here it goes..

Last August 29, 2008 on one of the RSS feeds of OTN TechBlog Sergio Leunissen posted a blog about Oracle Validated being available outside ULN which is very nice to hear.. then after 2 months (October), Alejandro Vargas posted a blog on how to do the Oracle-Validated installation on OEL5..

which then made me want to try it on OEL4, and is just in time because I want to shift to 64bit RAC on Linux (test environment on VMware)..

The whole installation is documented here: Oracle-Validated installation on OEL 4.5

Below are some Metalink Notes about Oracle Validated:

Linux OS Installation with Reduced Set of Packages for Running Oracle Database Server
Doc ID: Note:728346.1

Linux OS Installation with Reduced Set of Packages for Running Oracle Database Server without ULN/RHN
Doc ID: Note:579101.1

Defining a “default RPMs” installation of the Oracle Enterprise Linux (OEL) OS
Doc ID: Note:401167.1

Defining a “default RPMs” installation of the RHEL OS
Doc ID: Note:376183.1

Defining a “default RPMs” installation of the SLES OS
Doc ID: Note:386391.1

The ‘oracle-validated’ RPM Package for Installation Prerequisities
Doc ID: Note:437743.1

Below is the summary of the document:

The environment is a virtual machine with 1GB of RAM and two CPUs; the total time for the installation which includes media check, setting up networking, and additional RPMs was 30 minutes :) The good thing about this is the installation only consumed 1.5GB on the /usr filesystem and has a total of 583 RPMs, compared to my other installation (w/o using oracle-validated) which consumed around 2.5GB and has a total of 788 packages. Lesser packages will give you lesser services running on your system that makes the server easier to harden, manage, and maintain.
Although all the required RPMs are already installed on the server, you still have to create the needed directories and edit bash_profile, /etc/profile, and set the appropriate shmmax (/etc/sysctl.conf) value for your environment.

↧

Migrate from Windows XP 64bit to Ubuntu Intrepid Ibex 8.10 64bit

January 3, 2009, 2:04 am

≫ Next: Security, Forecasting Oracle Performance and Some stuff to post… soon…

≪ Previous: Oracle-Validated RPM on OEL 4.5

I’ve been using the newest and the greatest version of Ubuntu (Intrepid 8.10) for almost a month now and I’m happy with it :) Finally I’m using a Linux OS, and not even dual booting.. I’m an RHCT who’s aiming to be an RHCE by this first quarter of 2009.. So why not CentOS 5? Or other Linux distros?

Below are some of my opinions about it:

Ubuntu has a vast hardware support which does not need any kernel recompilation, building package from scratch, or any hardcore geek tweaks that you have to do in order to be happy with your machine. One example, if you’re in a meeting and you ask your co-worker, “that’s cool, can you project that slide for us?”.. he replies..”I can’t I’m running on CentOS 5, I haven’t worked on the module yet”.. Another is when you’re on a client and the only way to connect to their network is through Wireless and because your Wireless card is not supported you can’t get your job done..ouch!
I just want a user-friendly Linux desktop environment (not the server type) which will allow me to do similar things that I do on a Windows environment, also it must be backed by a very dynamic user community.. that makes the distribution keeps on evolving, and makes it very cool and exciting..
I have a 120GB external drive that has NTFS partitions, If I were to migrate to CentOS I would be needing NTFS-3g (NTFS driver) in order to use my external drive in read-write mode, apparently it’s not native on the CentOS distribution and there are just too many steps in order to make it work which is same case with the wireless card issue. When you install Ubuntu, you could just plug your external drive and immediately use it
Lastly, I also want to learn and be familiar with a Debian based distro..and I kinda like the Synaptic Package Manager, very easy to use with tons of packages available.. but Yum is still cool and powerful..

This is my laptop specs (NEO Elan L2110):

Technology Intel® Centrino® processor technology

Processor Intel® Core^TM2 Duo Processor T8100

Processor details 3 MB L2 cache, 2.10 Ghz, 800 Mhz FSB

Chipset Intel® GM965 + ICH8-M

Memory 4 GB DDR2

Graphics Intel® Graphics Media Accelerator X3100

Display 13.3″ WXGA LCD

Speakers 2 x High Fidelity Internal Speakers (1.5W)

Hard Disk Drive 160 GB SATA

Optical Disc Drive DVDRW SuperMulti Slot-in Type

Networking 10/100/1000 Mbps LAN

Wireless Intel® Wireless WiFi Link 4965AGN

Card Reader 4-in-1 Card Reader

Webcam 1.3 MegaPixels

Connectivity IEEE 1394, Bluetooth®

Battery 6-cell Lithium-ion Battery

Weight ~ 2.1 k

And here’s the step by step on how I did my migration:

On Windows:

1) Backup Calendar and Email (Thunderbird)

2) Backup all the files

3) Have an inventory of all the softwares installed

4) Backup bookmarks on Firefox

On Ubuntu:

5) Installation

Used the “alternate CD” then custom partition using LVM (if you are not using the “alternate CD” you can’t do LVM upon install), LVM is very flexible.. if ever I ran out of space on a certain file system I could just execute lvextend and resize2fs.. So I allotted the following:

120MB – BOOT,

2GB – SWAP,

1.5GB – ROOT,

138GB – HOME,

1GB – TMP,

4GB – USR,

1GB – VAR

For the keyboard layout I chose “USA” because I’m having problems with special characters when I chose the “USA International”
If you encounter the error “ESCOM” when you are hitting ENTER key when searching for a word on a man page, it is a keyboard bug.. just hit CTRL+M or CTRL+J for the workaround (https://bugs.launchpad.net/ubuntu/+source/xkeyboard-config/+bug/40905)
I used the 64bit Desktop Kernel of Ubuntu (although I can install the 64bit Server Kernel), why? Because I’m just running on a laptop, and I only have one SATA disk. If I install the server kernel it will have little or no difference on the performance. See the link (http://www.serverwatch.com/tutorials/article.php/3715071) for discussions on I/O scheduler, pre-emption, memory, ticks and HZ

6) After the installation, I just want to try if my VMs will run without problems on Ubuntu, my productivity depends on my VMs.. without them I can’t do all of my test cases for my clients, just follow the bullets below

Download VMware 1.08 – latest as of Jan 3, 2009, get the tar.gz version (http://www.vmware.com/download/server/)
Read the article (http://ubuntu-tutorials.com/2008/11/01/vmware-server-107-on-ubuntu-810-intrepid-2627-7-generic/) which you will be asked to download the http://www.insecure.ws/warehouse/vmware-update-2.6.27-5.5.7-2.tar.gz which is a patch that will update the vmware modules for the 2.6.27 kernel
Also read the (http://ubuntuforums.org/showthread.php?t=337040) for common problems that you may encounter when installing VMware on Ubuntu
If you’re having problems on the ENTER key inside your VMs, then read the link (http://communities.vmware.com/message/1091425), you have to append the xkeymap.nokeycodeMap = “true” on /etc/vmware/config file

7) Update Ubuntu (kernel and all packages)

Update your OS using the Synaptic Package Manager

8 ) Re-run the perl script vmware setup

Updating your OS will also update the kernel, VMware is kernel dependent so you have to rerun the “runme.pl” to recompile it to the new kernel

9) Install other softwares/packages

– Thunderbird and Lightning (for Email and Calendar)

– Filezilla (for easy file transfer)

– CHM viewer (for viewing my CHM ebooks)

– Kate (a powerful text editor with block selection, gedit has a column mode selection plugin but doesn’t really work)

– CrossOver (for using Windows Office 2k3, Visio, Project on a Linux environment.. I don’t have to worry about my Windows documents, got it free at http://www.codeweavers.com)

– K3B (a powerful burning tool)

– VLC player and other codecs

– VNC viewer and server

– cups-pdf (a PDF printer, solution to make it work: create PDF directory on the home directory, http://linux.derkeiler.com/Mailing-Lists/Ubuntu/2008-11/msg00745.html)

– Brasero (for creating ISO images, http://andieko.info/lang/en/how-to-create-iso-files-in-ubuntu/)

– KSnapshot (replacement for SnagIt, allows region selection)

– sysstat (system performance tools for linux) and ksh (korn shell).. I’ll be needing these to run OS watcher (by Oracle Support) on my machine..

– smbclient, samba-common, samba, libpam-smbpass (so you could share directories)

– sysv-rc-conf (SysV init runlevel config tool for the terminal, so you could have a “chkconfig” just like in RedHat)

– gpaint just like the paint program on Windows

– gwget a download manager which is a front-end for wget

10) Lastly, customize your desktop.. :)

This is the screenshot of my Desktop Environment..

You can see that I’m using my NTFS external drive and the following VMs:

2 node 10gR2 RAC on OEL4.4 x86 with OCFS2 and ASMlib
10gR2 on OEL4.5 x86 with ASM raw devices
10gR2 Data Guard on OEL 4.7 x86-64 both on ASMlib
Enterprise Manager Grid Control 10.2.0.4 on OEL 4.4 x86
10gR2 and 11g Database Vault and 10gR2 Audit Vault on OEL 4.4 x86

↧

Security, Forecasting Oracle Performance and Some stuff to post… soon…

February 27, 2009, 7:54 am

≫ Next: Single Instance and RAC Kernel/OS upgrade

≪ Previous: Migrate from Windows XP 64bit to Ubuntu Intrepid Ibex 8.10 64bit

I’ve been busy this February “playing around/studying” on the following:

1) Oracle Security products (Advance Security Option, Database Vault, Audit Vault, Data Masking, etc. etc.). Well, every organization must guard their digital assets against any threat (external/internal) because once compromised it could lead to negative publicity, lost revenue, litigation, lost of trust.. and the list goes on.. I’m telling you, Oracle has a lot to offer (breadth of products and features, some of them are even free!) on this area and you just need to have the knowledge to stitch them..

I’ll recommend a great book on Security, I believe everyone should have this (developers, DBAs, CxOs)..!
It’s a book by David Knox which is the chief engineer in Oracle’s Information Assurance Center (IAC). The IAC is a center of expertise that works with Oracle’s customers, partners, development, and consulting to design and develop security and high-availability solutions. Having a long time hands-on practice (working on Oracle security ever since) he was able to architect and implement solutions on organizations like United States Department of Defense and CIA.. and produce a book called “Effective Oracle Database 10g Security by Design”

2) The other one is Forecasting Oracle Performance… This is a book by Craig Shallahamer, a former Oracle employee, back then he’s with the System Performance Group (one of the largest and best collections of Oracle performance experts in the world) together with Cary Millsap (author of Optimizing Oracle Performance) and a few others..

Cary’s book is great, I’ve already read it and it will change the way you think about performance and tuning.. This is the only book that discusses Extended SQL trace in detail, the ins and outs and how to use it in different scenarios. This book also gave me a primer on what is Queueing Theory.. it tackles on the M/M/m Queueing Model but I felt it was more of an overview and not very detailed but at least I get the picture how useful it is..

Now presenting Craig’s work of art.. Forecasting Oracle Performance..

If you’re an Oracle professional, and you love complexities, and you like to be challenged, and you want to see whats ahead of you (in performance), and you love math (well I’m not really good at it, my brother is).. better read this!

Honestly I have two more performance/capacity planning book which I’ll still have to figure out how am I going to use it (specifically the formulas) on some of my performance tuning activities. These are:

Guerrilla Capacity Planning: A Tactical Approach to Planning for Highly Scalable Applications and Services
Performance by Design: Computer Capacity Planning By Example

Craig’s book is focused on the complex Oracle environment..and the book has contents like the following:

Learn how you manage your Service Levels or how to make Service Levels by using your workload samples,
at what workload and utilization you’ll enter the “elbow of the curve”
should you acquire, more CPUs (same speed)? or just replace it with faster CPUs (same number)? or both?
should you acquire, more disks (same speed)? or just replace it with faster disks (same number)? or both?
If there would be 200% additional workload, what will be the effect on response time?.. should I buy more CPUs? manage the workload? tune the application?
Learn basic statistics (Sample and Population, mean & weighted average, standard deviation, skew, histogram, inferences, residual analysis, confidence level and interval).. And you’ll learn to use the statement like.. 90% OF THE TIME, THE QUERY TAKES AN AVERAGE OF 3 SECONDS PLUS OR MINUS 2 SECONDS
Little’s law and Kendall’s notation
and many more….

You might be overwhelmed, but the forecasting work done in this book has methods and is tackled in a gradual manner. Chapter by Chapter you’ll see yourself progressing and not feeling the pain of understanding something that is beyond human cognition.. :) And Craig is a very good writer, he explains it in detail with easy to understand explanations (plus humor).. If you’re worrying about math, the formulas and models (essential forecasting, forecasting with Erlang C, ratio modeling, regression analysis, queueing theory) are easy to comprehend and use, you’ll get the hang of it once you use it..

Also, I was able to submit an errata on page 68 which Craig acknowledged (see image below).

We exchanged a couple of emails, and I asked a lot of questions about Chapter 4 (Basic Forecasting Statistics).. he answered it all and he was nice.. he said he had no idea I am from the Philippines.. and there will be an upcoming self published book about Oracle Performance Firefighting… that will be a good book.. Below is a scratch pad from Craig’s firefighting class:

3) And lastly.. are some of the stuff I’d like to post soon…

RAC kernel upgrade

Converting RAC to single instance

Block corruption issues, and how to resolve it

Word size change: 32bit to 64bit on 10.2.0.4

RAC pseudo standby database

↧

Single Instance and RAC Kernel/OS upgrade

April 4, 2009, 12:05 pm

≫ Next: OS Thread Startup

≪ Previous: Security, Forecasting Oracle Performance and Some stuff to post… soon…

This document will serve as a guide for the Kernel and OS upgrade activities for

Single Instance on ASM using raw devices
RAC with ASM (using ASMlib) and OCFS2

Upgrading the Kernel and OS is easy and will just need some few commands. The critical part is the dependencies once the Kernel gets updated, so if you’re using ASMlib and OCFS2 you’ll notice that after the upgrade they’re not working anymore… you can’t startup the ASM, then if your OCR and Voting Disk are on OCFS2 the CRS stack wont start all because the RPMs of ASMlib and OCFS2 are kernel dependent, also there are similar components/softwares that are kernel dependent so you have to check them out and do a risk analysis before doing the upgrade.

… If that’s the case, why do you upgrade your Kernel or OS? … Normally you don’t do this often on your production environment considering that you have 24/7 operations but if the business permits it’s ideal to make an upgrade policy on your OS/systems. Below are some of the “drivers why you need to upgrade”:

You need to upgrade the firmware of your SAN environment that requires you to be in a specific OS version

You need to upgrade the firmware of your HBA that needs a specific kernel version

You need to upgrade your RAID controller’s firmware that requires you to be in a specific kernel or OS version

Because the hardware/software vendor’s certification matrix is stating that you need to be in a specific kernel or OS version to be on a certified configuration

And many more…

So after reading this document, you know what to do/check once your boss/client/vendor tell you to upgrade the kernel or OS with an Oracle Database Single Instance or RAC running.

To view the document, click here

↧

OS Thread Startup

April 6, 2009, 12:42 pm

≫ Next: Advanced Oracle Troubleshooting by Tanel Poder in Singapore

≪ Previous: Single Instance and RAC Kernel/OS upgrade

Recently I encountered a performance problem scenario where a simple sqlplus “/ as sysdba” took about 2minutes to finish, this is critical to the client’s business because they have a local C program that loads Call Detail Reports on the database making use of local authentication for most of its operations and Sql*Loader to load the data, so this “2minutes of waiting” when accumulated greatly consumes significant time on their operations and greatly impacts the business.

When I arrived on the client I first checked the alert logs of both ASM (they have a separate home for ASM) and RDBMS, there were no errors…

Then I checked on the server to see if there were any CPU, IO, memory, swap, and network bottlenecks going on

The CPU run queue was zero and most of the time 90% idle

The disks were also most of the time idle

The memory utilization was low with 430MB free

Real memory Usage ----------------------------- Total Installed : 6.00000 GB Total Used : 5.57032 GB Total Free : 439.97264 MB REAL MEMORY USAGE : % : 92.800 -----------------------------

The paging on the server is not significant which is 1.1GB out of 16GB

Paging Space Usage ----------------------------- Total Pg Space : 16640.00000 MB Pg Space Used : 1187.77340 MB Percent Used PAGING : 7.100 -----------------------------

Then I did sqlplus “/ as sysdba” to do some SELECTs on some views (…yeah it really took about 2minutes…)

I looked at the free memory of the SGA memory structures… well, I think SGA is not the issue but I still have to find out…

select pool, round(bytes/1024/1024,2) size_mb from v$sgastat where name like '%free memory%'; POOL SIZE_MB ------------ ---------- shared pool 655.57 large pool 302.6 java pool 16 streams pool 15.99

At that time, there were only 4 users connected (that includes me) to the database and the C program was not running. I suspected that there are some contentions (latch) or concurrency problems that I’ll see on the wait interface… here’s a portion of the AWR report

   Elapsed:               60.11 (mins)
   DB Time:              121.50 (mins)

Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
os thread startup                     3,278       3,115    950   42.7 Concurrenc
latch free                            6,298       1,822    289   25.0      Other
flashback buf free by RVWR            1,771       1,721    972   23.6      Other
SQL*Net more data to client          60,087       1,574     26   21.6    Network
CPU time                                          1,088          14.9
          -------------------------------------------------------------

The DB Time was 3940.8 seconds (121.50 * 60) and the top event which is the “os thread startup” consumes 83% (3276/3940.8) of the total DB Time

And just for comparison, below is their normal workload which is generated on the same period (but different day) as the above

   Elapsed:               59.52 (mins)
   DB Time:               72.42 (mins)

Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
CPU time                                          2,009          46.2
db file scattered read              151,781         423      3    9.7   User I/O
db file sequential read              66,212         301      5    6.9   User I/O
latch: shared pool                      384          67    175    1.5 Concurrenc
log file sequential read                404          19     46    0.4 System I/O
          -------------------------------------------------------------

Hmmm so what is “os thread startup”? The ADDM report also outputs this event as a top finding

FINDING 1: 79% impact (3113 seconds)
------------------------------------
Wait event "os thread startup" in wait class "Concurrency" was consuming
significant database time.

   RECOMMENDATION 1: Application Analysis, 79% benefit (3113 seconds)
      ACTION: Investigate the cause for high "os thread startup" waits. Refer
         to Oracle's "Database Reference" for the description of this wait
         event. Use given SQL for further investigation.
      RATIONALE: The SQL statement with SQL_ID "4gd6b1r53yt88" was found
         waiting for "os thread startup" wait event.
         RELEVANT OBJECT: SQL statement with SQL_ID 4gd6b1r53yt88

   RECOMMENDATION 2: Application Analysis, 79% benefit (3113 seconds)
      ACTION: Investigate the cause for high "os thread startup" waits in
         Service "SYS$BACKGROUND".

   SYMPTOMS THAT LED TO THE FINDING:
      SYMPTOM: Wait class "Concurrency" was consuming significant database
               time. (80% impact [3156 seconds])

The event is not even documented on 8i, 9i, 10g, 11g official doc… I found some articles when I Google’d but they are not related to my issue… and Metalink does not have any helpful documents on this event for my situation…

There was already a Service Request regarding the issue, and the Support Engineer defined it as

“This wait event might be seen if the database server is executing on a platform that supports multi-threading. We enter this waiting state while a thread is starting up and leave the wait state when the thread has started or the startup request is cancelled.
This indicates some high contention at OS level avoiding even new process startup.
Issue is related to OS please involve system admin to solve same.”

The manager decided that we do a hard shutdown of the LPAR (we’re on P570 AIX 5.3) from the management console, and let’s see what it can do…

The DBA told me that once they are connected, all the operations are okay (DML, DDL, SQL*Loader).

So after the restart we tried to run the program for a while just to observe the behavior and what’s going on… apparently the local authentication is still slow!

For me, the challenge is how to have a response time profile of a session from the time it starts to connect with local authentication up to the time the session is connected so I would know what part of the operation is taking a lot of time.

What I remember, is when I was doing some queries to do some drill down on a session that was connecting using sqlplus “/ as sysdba” I see the background processes being spawned (QMNC, ARCn, etc..) are taking a lot of time in WAITING state with SECONDS_IN_WAIT reaching up to 20 seconds and has a wait event of “os thread startup”, then after a while the user gets connected then he’s now ready to do his thing without problems…Meaning after he’s connected there are no more complains…

SQL> select w.sid, s.sql_hash_value, s.program, w.event, w.wait_time/100 t, w.seconds_in_wait seconds_in_wait, w.state, w.p1, w.p2, w.p3
  2  from v$session s, v$session_wait w
  3  where s.sid = w.sid
  4  and w.state = 'WAITING'
  5  and w.event like '%os thread startup%'
  6  order by 6 asc;

   SID SQL_HASH_VALUE PROGRAM                 EVENT                       T SECONDS_IN_WAIT STATE                     P1         P2         P3
------ -------------- ----------------------- ------------------ ---------- --------------- ----------------- ---------- ---------- ----------
   121              0 oracle@ps570roc01 (QMNC os thread startup)          0              18 WAITING                    0          0          0

To know what’s going on when a user connects and for me to account the operation where it takes significant time; I created a database user TRACE_USER and created a logon trigger that invokes Extended SQL Trace Level 12. This will give me the timing information that I need when the user is connecting to the database.

To create the user:

create user trace_user identified by trace_user account unlock; grant create session,dba to trace_user;

To create the logon trigger:

create or replace trigger sys.set_trace
after logon on database
when (user = 'trace_user')
declare
lcommand varchar(200);
begin
execute immediate 'alter session set statistics_level=all';
execute immediate 'alter session set max_dump_file_size=unlimited';
execute immediate 'alter session set events ''10046 trace name context forever, level 12''';
end set_trace;
/

After the user and the trigger are created, logon as the TRACE_USER by doing the following command:

sqlplus "trace_user/trace_user" << eof exit; eof

This will generate an Extended SQL Trace (10046) trace file under the $ORACLE_BASE/admin/<db_name>/udump directory

Then see if there’s significant time on the database calls (PARSE,BIND,EXEC,FETCH) by looking at the “e=” section which means elapsed duration consumed by the database call

And check if there’s significant time on wait events by looking at the “ela=” section which means elapsed duration consumed by the wait event (probably by doing a system call)

Below is a small portion of the trace file generated, we are interested on the highlighted portions…

PARSING IN CURSOR #5 len=131 dep=1 uid=0 oct=3 lid=0 tim=15687899202263 hv=1389591971 ad='ca9a7948'
select privilege#,nvl(col#,0),max(mod(nvl(option$,0),2))from objauth$ where obj#=:1 and grantee#=:2 group by privilege#,nvl(col#,0)
END OF STMT
EXEC #5:c=0,e=145,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,tim=15687899202260
WAIT #5: nam='db file sequential read' ela= 30005 file#=1 block#=24208 blocks=1 obj#=-1 tim=15687899232346
FETCH #5:c=0,e=30075,p=1,cr=2,cu=0,mis=0,r=0,dep=1,og=4,tim=15687899232396

There were no database calls that consumed significant time (numbers below are microseconds, that’s for 9i and above):

less oracle_ora_1118270.trc | grep -i ",e=" | cut -d , -f2 > karlarao.txt ; sed -n 's/e=/ /p' karlarao.txt | sort -nr | less

 358746
 183162
 61293
 44661
 32580
 30075
 28695
 26950
 25837
 24244
 23519
 20543
 20132
 19449
 19290
 18809
 18250
 17230
 16507
 16349
 13438
 11716
 11690
 11284
 10137

There were no wait events that consumed significant time (numbers below are microseconds, that’s for 9i and above):

less oracle_ora_1118270.trc \| grep -i "ela=" \| cut -d " " -f8 \| sort -nr \| less 30005 28624 13253 11592 9650

There was no “os thread startup” event!

less oracle_ora_1118270.trc | grep -i "ela=" | cut -d "=" -f2 | uniq

'SQL*Net message to client' ela
'SQL*Net message from client' ela
'SQL*Net message to client' ela
'SQL*Net message from client' ela
'SQL*Net message to client' ela
'SQL*Net message from client' ela
'SQL*Net message to client' ela
'SQL*Net message from client' ela
'db file sequential read' ela
'SQL*Net message to client' ela
'SQL*Net message from client' ela
'db file sequential read' ela
'SQL*Net message to client' ela
'db file sequential read' ela
'SQL*Net message from client' ela
'SQL*Net message to client' ela
'SQL*Net message from client' ela
'db file sequential read' ela
'SQL*Net message to client' ela
'SQL*Net message from client' ela
'SQL*Net message to client' ela
'SQL*Net message from client' ela
'SQL*Net message to client' ela
'SQL*Net message from client' ela
'SQL*Net message to client' ela
'SQL*Net message from client' ela

I need to have some instrumentation on the system calls, so I’ll use a UNIX tool,

presenting… TRUSS…

This tool will let you do some application tracing, for us we are interested in tracing SQL*Plus, and display the calls that an application makes to external libraries and the kernel… sounds cool :)

A “man truss” will give you an overview, but if you want more detail there are some good references at the IBM site or Metalink

Execute the following commands (one at a time):

truss -c sqlplus "/ as sysdba" << EOF exit; EOF truss -d sqlplus "/ as sysdba" << EOF exit; EOF

The “-c” switch counts traced system calls, faults, and signals

The “-d” switch displays the timestamp in seconds on each line. The first line of the trace output will show the base time from which the individual time stamps are measured

Bwahaha, Now I know the problem!

The DNS service was causing the slow down on the sqlplus “/ as sysdba” operation and also the high “os thread startup” waits

The text below is the portion of “truss” output where the slow down occurs denoted by “(sleeping…)” events when reading on the /etc/resolv.conf and /etc/hosts. Please see the highlighted region below

ENOTTY and ECONNREFUSED are the errors that you’ll see, you’ll find more details on the error messages on this file /usr/include/sys/errno.h
Below are their meanings

#define ENOTTY 25 /* Inappropriate I/O control operation / #define ECONNREFUSED 79 / Connection refused */

The portion of “truss -d” where the slow down occurs, you’ll see a leap on the timing information from 2.3379 seconds (reading /etc/resolv.conf) –> 7.3477 seconds –> 17.3489 seconds –> 37.3555 seconds (reading /etc/hosts) –> 70.3863 seconds (already connected)

The timing information below may not exactly match the real “time” that the user is waiting, it’s because the SQL*Plus is being probed to know the part of the underlying system calls where there are significant waits

0.3172:        gethostname(0x0FFFFFFFFFFF9920, 256) = 0
0.3175:        _getpid()                        = 1102056
0.3177:        open("/etc/resolv.conf", O_RDONLY) = 8
0.3180:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
0.3182:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
kread(8, " d o m a i n     this is the domain name, i removed it".., 4096)    = 69
kread(8, " d o m a i n     this is the domain name, i removed it".., 4096)    = 0
0.3190:        close(8)                         = 0
0.3194:        socket(2, 2, 0)                  = 8
0.3197:        getsockopt(8, 65535, 4104, 0x0FFFFFFFFFFF70E4, 0x0FFFFFFFFFFF70E0) = 0
0.3199:        connext(8, 0x09001000A0018600, 16) = 0
0.3202:        send(8, 0x0FFFFFFFFFFF7FF0, 41, 0) = 41
0.3205:        _poll(0x0FFFFFFFFFFF7190, 1, 5000) = 1
0.3208:        nrecvfrom(8, 0x0FFFFFFFFFFF9320, 1024, 0, 0x0FFFFFFFFFFF7950, 0x0FFFFFFFFFFF7178) Err#79 ECONNREFUSED
0.3210:        close(8)                         = 0
0.3212:        socket(2, 2, 0)                  = 8
0.3215:        sendto(8, 0x0FFFFFFFFFFF7FF0, 41, 0, 0x09001000A0018610, 16) = 41
0.3217:        _poll(0x0FFFFFFFFFFF7190, 1, 5000) = 1
0.3220:        nrecvfrom(8, 0x0FFFFFFFFFFF9320, 1024, 0, 0x0FFFFFFFFFFF7950, 0x0FFFFFFFFFFF7178) = 108
0.3222:        close(8)                         = 0
0.3224:        socket(2, 2, 0)                  = 8
0.3227:        getsockopt(8, 65535, 4104, 0x0FFFFFFFFFFF70E4, 0x0FFFFFFFFFFF70E0) = 0
0.3229:        connext(8, 0x09001000A0018600, 16) = 0
0.3231:        send(8, 0x0FFFFFFFFFFF7FF0, 28, 0) = 28
0.3233:        _poll(0x0FFFFFFFFFFF7190, 1, 5000) = 1
0.3236:        nrecvfrom(8, 0x0FFFFFFFFFFF9320, 1024, 0, 0x0FFFFFFFFFFF7950, 0x0FFFFFFFFFFF7178) Err#79 ECONNREFUSED
0.3238:        close(8)                         = 0
0.3240:        socket(2, 2, 0)                  = 8
0.3243:        sendto(8, 0x0FFFFFFFFFFF7FF0, 28, 0, 0x09001000A0018610, 16) = 28
0.3245:        _poll(0x0FFFFFFFFFFF7190, 1, 5000) = 1
0.3248:        nrecvfrom(8, 0x0FFFFFFFFFFF9320, 1024, 0, 0x0FFFFFFFFFFF7950, 0x0FFFFFFFFFFF7178) = 28
0.3250:        close(8)                         = 0
0.3252:        socket(2, 2, 0)                  = 8
0.3373:        sendto(8, 0x0FFFFFFFFFFF7FF0, 28, 0, 0x09001000A0018600, 16) = 28
2.3379:        _poll(0x0FFFFFFFFFFF7190, 1, 5000) (sleeping...)
2.3379:        _poll(0x0FFFFFFFFFFF7190, 1, 5000) = 0
5.3468:        close(8)                         = 0
5.3470:        socket(2, 2, 0)                  = 8
5.3472:        sendto(8, 0x0FFFFFFFFFFF7FF0, 28, 0, 0x09001000A0018600, 16) = 28
7.3477:        _poll(0x0FFFFFFFFFFF7190, 1, 10000) (sleeping...)
7.3477:        _poll(0x0FFFFFFFFFFF7190, 1, 10000) = 0
15.3479:        close(8)                        = 0
15.3482:        socket(2, 2, 0)                 = 8
15.3484:        sendto(8, 0x0FFFFFFFFFFF7FF0, 28, 0, 0x09001000A0018600, 16) = 28
17.3489:        _poll(0x0FFFFFFFFFFF7190, 1, 20000) (sleeping...)
17.3489:        _poll(0x0FFFFFFFFFFF7190, 1, 20000) = 0
35.3491:        close(8)                        = 0
35.3495:        getdomainname(0x0FFFFFFFFFFF9480, 256) = 0
35.3497:        open("/etc/hosts", O_RDONLY)    = 8
35.3500:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
35.3502:        kfcntl(8, F_SETFD, 0x0000000000000001) = 0
35.3505:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
kread(8, " #   @ ( # ) 4 7\t 1 . 1".., 4096)    = 3453
kread(8, " #   @ ( # ) 4 7\t 1 . 1".., 4096)    = 0
35.3514:        close(8)                        = 0
kwrite(9, "01 �0603 s".., 432)    = 432
37.3555:        kread(10, "16060280".., 2064) (sleeping...)
kread(10, "06 E06\b".., 2064)   = 1605
70.3762:        open("/oracle/app/product/10.2/db_1/rdbms/mesg/oraus.msb", O_RDONLY) = 8
70.3764:        kfcntl(8, F_SETFD, 0x0000000000000001) = 0
70.3767:        lseek(8, 0, 0)                  = 0
kread(8, "1513 "011303\t\t".., 256)     = 256
70.3772:        lseek(8, 512, 0)                = 512
kread(8, "1A Q 5 C [ V u �85 �9480".., 512)     = 512
70.3776:        lseek(8, 1024, 0)               = 1024
kread(8, "18 $ 4 = G S".., 512)     = 512
70.3781:        lseek(8, 50688, 0)              = 50688
kread(8, "\f05 ] P05 ^".., 512)     = 512
70.3786:        lseek(8, 512, 0)                = 512
kread(8, "1A Q 5 C [ V u �85 �9480".., 512)     = 512
70.3790:        lseek(8, 1024, 0)               = 1024
kread(8, "18 $ 4 = G S".., 512)     = 512
70.3795:        lseek(8, 50688, 0)              = 50688
kread(8, "\f05 ] P05 ^".., 512)     = 512
70.3799:        lseek(8, 512, 0)                = 512
kread(8, "1A Q 5 C [ V u �85 �9480".., 512)     = 512
70.3804:        lseek(8, 1024, 0)               = 1024
kread(8, "18 $ 4 = G S".., 512)     = 512
70.3808:        lseek(8, 50688, 0)              = 50688
kread(8, "\f05 ] P05 ^".., 512)     = 512
70.3813:        lseek(8, 512, 0)                = 512
kread(8, "1A Q 5 C [ V u �85 �9480".., 512)     = 512
70.3817:        lseek(8, 1024, 0)               = 1024
kread(8, "18 $ 4 = G S".., 512)     = 512
70.3822:        lseek(8, 51712, 0)              = 51712
kread(8, "\n0589 D058A".., 512)     = 512
70.3827:        close(8)                        = 0
kwrite(9, " <0611 k".., 60)     = 60
kread(10, " �06\b".., 2064)   = 179
kwrite(9, " U0603 h".., 85)     = 85
kread(10, "1606\b".., 2064)   = 22
kwrite(9, " U0603 h".., 85)     = 85
kread(10, "1606\b".., 2064)   = 22
70.3854:        lseek(4, 512, 0)                = 512
kread(4, "17 �".., 512)     = 512
70.3858:        lseek(4, 1024, 0)               = 1024
kread(4, "16 * R h819E".., 512)     = 512
70.3863:        lseek(4, 4096, 0)               = 4096
kread(4, "\f82 P86".., 512)     = 512

kwrite(1, "\n", 1)                              = 1
Connected to:
kwrite(1, " C o n n e c t e d   t o".., 14)     = 14
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
kwrite(1, " O r a c l e   D a t a b".., 77)     = 77
With the Partitioning, OLAP, Data Mining and Real Application Testing options
kwrite(1, " W i t h   t h e   P a r".., 78)     = 78

kwrite(1, "\n", 1)                              = 1
70.3882:        kfcntl(1, F_GETFL, 0x0000000000000008) = 2
70.3886:        __libc_sbrk(0x0000000000030020) = 0x000000001023C880
70.3892:        access("login.sql", 0)          Err#2  ENOENT
70.3895:        access("/oracle/app/product/10.2/db_1/sqlplus/admin/glogin.sql", 0) = 0
70.3898:        statfs("/oracle/app/product/10.2/db_1/sqlplus/admin/glogin.sql", 0x0FFFFFFFFFFFDA10) = 0

Apparently there was really a problem with the DNS when it was checked by the System Administrator. He just altered some network related files and instantly we checked on the local authentication… and we could quickly do a local authentication again! Whooho! and everybody was happy :)

Below is the good “truss” output of the local authentication, which takes about 3 seconds from the timing information

3.0713:        gethostname(0x0FFFFFFFFFFF9900, 256) = 0
3.0716:        _getpid()                        = 2539762
3.0719:        open("/etc/resolv.conf", O_RDONLY) = 8
3.0721:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
3.0724:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
kread(8, " d o m a i n     this is the domain name, i removed it".., 4096)    = 70
kread(8, " d o m a i n     this is the domain name, i removed it".., 4096)    = 0
3.0731:        close(8)                         = 0
3.0735:        socket(2, 2, 0)                  = 8
3.0740:        getsockopt(8, 65535, 4104, 0x0FFFFFFFFFFF70C4, 0x0FFFFFFFFFFF70C0) = 0
3.0742:        connext(8, 0x09001000A0018600, 16) = 0
3.0746:        send(8, 0x0FFFFFFFFFFF7FD0, 41, 0) = 41
3.0750:        _poll(0x0FFFFFFFFFFF7170, 1, 5000) = 1
3.0753:        nrecvfrom(8, 0x0FFFFFFFFFFF9300, 1024, 0, 0x0FFFFFFFFFFF7930, 0x0FFFFFFFFFFF7158) = 108
3.0756:        send(8, 0x0FFFFFFFFFFF7FD0, 28, 0) = 28
3.0758:        _poll(0x0FFFFFFFFFFF7170, 1, 5000) = 1
3.0761:        nrecvfrom(8, 0x0FFFFFFFFFFF9300, 1024, 0, 0x0FFFFFFFFFFF7930, 0x0FFFFFFFFFFF7158) = 28
3.0764:        close(8)                         = 0
3.0767:        getdomainname(0x0FFFFFFFFFFF9460, 256) = 0
3.0769:        open("/etc/hosts", O_RDONLY)     = 8
3.0771:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
3.0774:        kfcntl(8, F_SETFD, 0x0000000000000001) = 0
3.0777:        kioctl(8, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
kread(8, " #   @ ( # ) 4 7\t 1 . 1".., 4096)    = 3453
kread(8, " #   @ ( # ) 4 7\t 1 . 1".., 4096)    = 0
3.0786:        close(8)                         = 0
kwrite(9, "01 �0603 s".., 432)    = 432
kread(10, "06 H06\b".., 2064)   = 1608
3.0973:        open("/oracle/app/product/10.2/db_1/rdbms/mesg/oraus.msb", O_RDONLY) = 8
3.0975:        kfcntl(8, F_SETFD, 0x0000000000000001) = 0
3.0978:        lseek(8, 0, 0)                   = 0
kread(8, "1513 "011303\t\t".., 256)     = 256
3.0983:        lseek(8, 512, 0)                 = 512
kread(8, "1A Q 5 C [ V u �85 �9480".., 512)     = 512
3.0988:        lseek(8, 1024, 0)                = 1024
kread(8, "18 $ 4 = G S".., 512)     = 512
3.0993:        lseek(8, 50688, 0)               = 50688
kread(8, "\f05 ] P05 ^".., 512)     = 512
3.0997:        lseek(8, 512, 0)                 = 512
kread(8, "1A Q 5 C [ V u �85 �9480".., 512)     = 512
3.1002:        lseek(8, 1024, 0)                = 1024
kread(8, "18 $ 4 = G S".., 512)     = 512
3.1006:        lseek(8, 50688, 0)               = 50688
kread(8, "\f05 ] P05 ^".., 512)     = 512
3.1011:        lseek(8, 512, 0)                 = 512
kread(8, "1A Q 5 C [ V u �85 �9480".., 512)     = 512
3.1015:        lseek(8, 1024, 0)                = 1024
kread(8, "18 $ 4 = G S".., 512)     = 512
3.1021:        lseek(8, 50688, 0)               = 50688
kread(8, "\f05 ] P05 ^".., 512)     = 512
3.1025:        lseek(8, 512, 0)                 = 512
kread(8, "1A Q 5 C [ V u �85 �9480".., 512)     = 512
3.1029:        lseek(8, 1024, 0)                = 1024
kread(8, "18 $ 4 = G S".., 512)     = 512
3.1034:        lseek(8, 51712, 0)               = 51712
kread(8, "\n0589 D058A".., 512)     = 512
3.1038:        close(8)                         = 0
kwrite(9, " <0611 k".., 60)     = 60
kread(10, " �06\b".., 2064)   = 179
kwrite(9, " U0603 h".., 85)     = 85
kread(10, "1606\b".., 2064)   = 22
kwrite(9, " U0603 h".., 85)     = 85
kread(10, "1606\b".., 2064)   = 22
3.1064:        lseek(4, 512, 0)                 = 512
kread(4, "17 �".., 512)     = 512
3.1069:        lseek(4, 1024, 0)                = 1024
kread(4, "16 * R h819E".., 512)     = 512
3.1073:        lseek(4, 4096, 0)                = 4096
kread(4, "\f82 P86".., 512)     = 512

kwrite(1, "\n", 1)                              = 1
Connected to:
kwrite(1, " C o n n e c t e d   t o".., 14)     = 14
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
kwrite(1, " O r a c l e   D a t a b".., 77)     = 77
With the Partitioning, OLAP, Data Mining and Real Application Testing options
kwrite(1, " W i t h   t h e   P a r".., 78)     = 78

kwrite(1, "\n", 1)                              = 1
3.1095:        kfcntl(1, F_GETFL, 0x0000000000000008) = 2
3.1099:        __libc_sbrk(0x0000000000030020)  = 0x000000001023C880
3.1105:        access("login.sql", 0)           Err#2  ENOENT
3.1109:        access("/oracle/app/product/10.2/db_1/sqlplus/admin/glogin.sql", 0) = 0
3.1112:        statfs("/oracle/app/product/10.2/db_1/sqlplus/admin/glogin.sql", 0x0FFFFFFFFFFFD9F0) = 0

↧

Advanced Oracle Troubleshooting by Tanel Poder in Singapore

April 19, 2009, 11:23 pm

≫ Next: Understanding the SCN

≪ Previous: OS Thread Startup

When I first saw that Tanel will conduct his seminar in Singapore, I told myself that I would even spend my own money just to be on that training! I’ve already read performance books like Optimizing Oracle Performance, Oracle 8i Internal Services, Forecasting Oracle Performance… And after that I still want more, and I still have questions that need to be answered. Well, if you’re on a tight budget you just opt to download some more docs/books to do multiple reads coupled with research/test cases and also reading through others blog…
But thanks to my boss for the funding, I was there! :-)

The seminar has 8 chapters,

1) End to end lifecycle of a database request (33 slides)
2) Oracle, OS, and hardware interaction (39 slides)
3) Oracle kernel core services (39 slides)
4) OWI and kernel instrumentation framework (34 slides)
5) Lifecycle of a cursor (27 slides)
6) SQL execution row source operators and data flow (34 slides)
7) Logical and Physical IO internals (37 slides)
8 ) Lifecycle of a transaction (32 slides)

On the first day we were able to tackle the three chapters…

The 1st was about how to “Systematically Troubleshoot” performance problems. That Oracle is just a program and every execution that is happening is not magic, using the right tool for the right problem you’ll be able to pinpoint the root cause. Also he emphasized on how important it is to make the user’s session a starting point on diagnosis and he demo’d on how to make use of v$system_wait, snapper, ASH, v$sesstat, and of course if the built-in instrumentation is not enough… the process stack sampling is very useful that will tell you what kernel function where most of the time is spent…he also taught us how to read the stack tree using that magic Metalink note (175982.1) ;)

2nd, dives into the OS troubleshooting, how you’ll systematically approach memory and CPU starvation on both Oracle and OS drilling down to the processes using Oracle (heapdump) and OS tools (ps, pstack, strace, procmon.exe for Windows – yeah you could also do process stack samples, pmap, top, vmstat)

3rd, was more of the internals where he discussed the core services of Oracle Kernel starting with the KGH module, the shared pool, and some more heaps, heap dumps and state objects, some X$ tables, and of course the most important part of the chapter… the Latching. He demo’d on how to systematically troubleshoot latching problems using latchprof, latchprofx, and snapper.

Then… At the end of the day, I was able to discuss with Tanel the concept of “performance visualization” see links below. I mean, yeah you can do reactive tuning and be very good at internals… but it doesn’t stop there… you have to be proactive, and visualizing/trending your performance numbers will help you on that. I’ve read his paper with Neil Gunther about how Multidimensional Visualization helps on analysing Oracle session level performance data that sometimes the usual x-y charts could mislead. He showed me some cool 3d performance monitoring tools running on JVM which is still on the works, but it’s really cool ;)

And we also discussed about “capacity planning” because Neil Gunther is the author of the “Super Serial Scalability model” which he worked with (on the Visualization CMG paper) and Craig Shallahamer discussed it on the Chapter 10 (Scalability) of his book which is very interesting. Well, Tanel will still have to find out how efficient or realistic the “forecasting models” will be when applied with Oracle performance data. Well me too! But it still trickles down to response time and visualization ;)

The second day…

We started with the Oracle Wait Interface, he answered my question on how would you know if it’s a syscall? Usercall? Or kernel function call?… he just used pstack, pmap, and the magic Metalink note. Cool ;) Then he explained why the v$session is the most important view in 10g above. He showed us how to pinpoint if it’s a pl/sql that is executing or just plain SQL; yeah he has scripts for that! … the “sample” script is very useful for finding out which SQLs are problematic. He ended the chapter with identifying the session whether it is client-server or N-tiered application and demo’d on how to make use of dbms_session and dbms_monitor and how to trace these sessions.

5th discussed about lifecycle of cursors. Tanel discussed the internals of Library Cache and how applications live and die by how you do cursor management ;) He also got good scripts here!

6th for me is the most interesting, this is where you start to glue together the info about the Oracle and OS interaction from day1 that Oracle is just a big complex C program… he started the chapter by clearly explaining how to properly get and read the explain plan (root -> branch -> leaves and row source -> filters -> access path) and associating each line with the Oracle kernel functions which you could get by doing a process stack and feeding the output to his “os_explain” tool…cool ;) Then he ended the chapter with SQL plan profiling using some of his formatted explain plan scripts…

Unfortunately we ran out of time… we still have two chapters left! Tanel have to catch a flight to Sydney at 8pm. But he told us he will be blogging more about the later chapters, and at least we have his module… So the last demo was his “perfsheet” ;) a very good visualization tool. Nice job with the pivot tables and 3d charts!

All in all, the popular geek from Estonia showed me how Oracle really works! (well, you owe us two chapters :p)

BTW, Porus Homi Havewala (http://enterprise-manager.blogspot.com) was also on the class…

Below are some of the seminar pictures:

Below are some links about Visualization:

Neil Gunther & Tanel Poder – Multidimensional Visualization of Oracle Performance using Barry007
http://arxiv.org/pdf/0809.2532

Tanel Poder’s Visualization Tool
http://blog.tanelpoder.com/2008/12/28/performance-visualization-made-easy-perfsheet-20-beta/
http://blog.tanelpoder.com/2009/02/14/performance-visualization-capacity-planning-and-hotsos-symposium/

Neil Gunther – A.A. Michelson Award Acceptance Speech
http://www.perfdynamics.com/Test/aamspeech.pdf

Neil Gunther’s Blog
http://perfdynamics.blogspot.com/2008/12/gunther-receives-aa-michelson-award.html

↧

Understanding the SCN

May 8, 2009, 2:25 am

≫ Next: Yast on OEL

≪ Previous: Advanced Oracle Troubleshooting by Tanel Poder in Singapore

For the DBAs who want to have a refreser on SCN (system change number), this article article is very nice and explained clearly written by Sandeep Makol. It started on where you ‘ll find info for SCN (controlfile and datafile headers) then goes to the backup and recovery scenarios where knowledge of this “magic number” is very useful.

Below are some useful scripts (with sample output) as well

-- get specific datafile
col name format a50
select 'controlfile' "SCN location",'SYSTEM checkpoint' name,checkpoint_change#
from v$database
union
select 'file in controlfile',name,checkpoint_change#
from v$datafile where lower(name) like '%&&datafile%'
union
select 'file header',name,checkpoint_change#
from v$datafile_header where lower(name) like '%&&datafile%';

SCN location	    NAME					       CHECKPOINT_CHANGE#
------------------- -------------------------------------------------- ------------------
controlfile	    SYSTEM checkpoint						  1151614
file header	    +DATA_1/ivrs/datafile/sysaux.258.652821943			  1151614
file header	    +DATA_1/ivrs/datafile/system.267.652821909			  1151614
file header	    +DATA_1/ivrs/datafile/system_02.dbf 			  1151614
file header	    +DATA_1/ivrs/datafile/undotbs1.257.652821933		  1151614
file header	    +DATA_1/ivrs/datafile/users.263.652821963			  1151614
file in controlfile +DATA_1/ivrs/datafile/sysaux.258.652821943			  1151614
file in controlfile +DATA_1/ivrs/datafile/system.267.652821909			  1151614
file in controlfile +DATA_1/ivrs/datafile/system_02.dbf 			  1151614
file in controlfile +DATA_1/ivrs/datafile/undotbs1.257.652821933		  1151614
file in controlfile +DATA_1/ivrs/datafile/users.263.652821963			  1151614

-- get distinct checkpoint_change#
select checkpoint_change#, 'SYSTEM checkpoint in controlfile' "SCN location"
from v$database
union
select distinct checkpoint_change#, 'file in controlfile'
from v$datafile
union
select distinct checkpoint_change#, 'file header'
from v$datafile_header;

CHECKPOINT_CHANGE# SCN location
------------------ --------------------------------
	   1151614 SYSTEM checkpoint in controlfile
	   1151614 file header
	   1151614 file in controlfile

-- get distinct datafile count
select 'controlfile' "SCN location",'SYSTEM checkpoint' name,checkpoint_change#
from v$database
union
select 'file in controlfile',to_char(count(*)),checkpoint_change#
from v$datafile
group by checkpoint_change#
union
select 'file header',to_char(count(*)),checkpoint_change#
from v$datafile_header
group by checkpoint_change#;

SCN location	    NAME					       CHECKPOINT_CHANGE#
------------------- -------------------------------------------------- ------------------
controlfile	    SYSTEM checkpoint						  1151614
file header	    5								  1151614
file in controlfile 5								  1151614

-- info from x$kcvfh (All file headers)
SELECT hxfil file_num,substr(hxfnm,1,40) file_name,fhtyp type,hxerr validity, fhscn chk_ch#, fhtnm tablespace_name,fhsta status,fhrba_seq sequence
FROM x$kcvfh;

  FILE_NUM FILE_NAME					  TYPE	 VALIDITY CHK_CH#	   TABLESPACE_NAME		      STATUS   SEQUENCE
---------- ---------------------------------------- ---------- ---------- ---------------- ------------------------------ ---------- ----------
	 1 +DATA_1/ivrs/datafile/system.267.6528219	     3		0 1151614	   SYSTEM				8196	    146
	 2 +DATA_1/ivrs/datafile/undotbs1.257.65282	     3		0 1151614	   UNDOTBS1				   4	    146
	 3 +DATA_1/ivrs/datafile/sysaux.258.6528219	     3		0 1151614	   SYSAUX				   4	    146
	 4 +DATA_1/ivrs/datafile/users.263.65282196	     3		0 1151614	   USERS				   4	    146
	 5 +DATA_1/ivrs/datafile/system_02.dbf		     3		0 1151614	   SYSTEM				   4	    146

↧

Yast on OEL

May 8, 2009, 1:04 pm

≫ Next: Diagnosing and Resolving “gc block lost”

≪ Previous: Understanding the SCN

Lately I’ve been playing with Enterprise Manager Grid Control 10.2.0.4 on OEL 4.4 (I’ll upgrade this to 10.2.0.5 soon) and I’ve had a couple of product presentation and demo focusing on the Database Enterprise Management, well among all the packs under this Grid Control “Area” the hottest are the Configuration Management and Data Masking.

…While clicking around the huge monitoring tool I’ve reached the page of “Targets -> Hosts -> Administration”…that happens to be part of the Oracle Management Pack for Linux and noticed it was not working and says it needs the “Yast wrapper available at oss.oracle.com” hmm maybe not the exact message but it lead me to download this file http://oss.oracle.com/projects/yast/dist/files/el4/20070117/yast_el4_x86_32.tar which is a port of Yast to Oracle Enterprise Linux. COOL! :-)

Well Yast (Yet another Setup Tool) is a pretty comprehensive tool which allows you to install software, configure hardware, set up networks and servers, and more…
But I’m a RedHat guy and I’m used to the way I manually do all these stuffs using the native utilities. (So even if I have this cool tool installed on my machine maybe I will not be able to use it :p )

Here’s the step by step guide to get that cool thing running:

1) Install EMGC Release 4
2) Download YAST plugin at http://oss.oracle.com/projects/yast
3) Untar and install the plugin, see below

[root@emgc1 yast_el4_x86_32]# ./install.sh 

Now checking the necessary rpms...........
Checking alsa-lib ....... found
Checking alsa-utils ....... found
Checking ash ....... found
Checking beecrypt ....... found
Checking bind-utils ....... found
Checking bzip2-libs ....... found
Checking compat-libstdc++-33 ....... found
Checking cracklib ....... found
Checking cups-libs ....... found
Checking curl ....... found
Checking e2fsprogs ....... found
Checking elfutils-libelf ....... found
Checking filesystem ....... found
Checking glibc ....... found
Checking glibc-common ....... found
Checking gzip ....... found
Checking hdparm ....... found
Checking info ....... found
Checking initscripts ....... found
Checking krb5-libs ....... found
Checking libgcc ....... found
Checking libidn ....... found
Checking libselinux ....... found
Checking libstdc++ ....... found
Checking libxml2 ....... found
Checking logrotate ....... found
Checking ncurses ....... found
Checking net-tools ....... found
Checking openldap ....... found
Checking openssl ....... found
Checking parted ....... found
Checking pciutils ....... found
Checking perl ....... found
Checking perl-Digest-SHA1 ....... found
Checking popt ....... found
Checking python ....... found
Checking qt ....... found
Checking rhpl ....... found
Checking rpm ....... found
Checking rpm-libs ....... found
Checking sysfsutils ....... found
Checking zlib ....... found

Now checking the necessary tools...........
Checking  /bin/bash ....... found
Checking  /bin/ln ....... found
Checking  /bin/mktemp ....... found
Checking  /bin/rm ....... found
Checking  /bin/sed ....... found
Checking  /bin/sh ....... found
Checking  /sbin/ldconfig ....... found
Checking  /usr/bin/env ....... found
Checking  /usr/bin/find ....... found
Checking  /usr/bin/host ....... found
Checking  /usr/bin/perl ....... found
Checking  /usr/bin/python ....... found
Checking  /usr/bin/xargs ....... found
Checking  /usr/sbin/lsof ....... found

Environment is OK. Now begin installing yast.........

Preparing...                ########################################### [100%]
   1:fillup                 ########################################### [100%]
Preparing...                ########################################### [100%]
   1:aaa_base               ########################################### [100%]
Updating etc/sysconfig/language...
Updating etc/sysconfig/backup...
Updating etc/sysconfig/boot...
Updating etc/sysconfig/displaymanager...
Updating etc/sysconfig/kernel...
Updating etc/sysconfig/suseconfig...
Updating etc/sysconfig/clock...
Updating etc/sysconfig/proxy...
Updating etc/sysconfig/windowmanager...
Updating etc/sysconfig/sysctl...
Updating etc/sysconfig/java...
Updating etc/sysconfig/mail...
Updating etc/sysconfig/ide...
Updating etc/sysconfig/cron...
Updating etc/sysconfig/news...
Preparing...                ########################################### [100%]
   1:liby2util              ########################################### [100%]
Preparing...                ########################################### [100%]
   1:netcat                 ########################################### [100%]
Preparing...                ########################################### [100%]
   1:libxcrypt              ########################################### [100%]
Preparing...                ########################################### [100%]
   1:insserv                ########################################### [100%]
Preparing...                ########################################### [100%]
   1:hwinfo                 ########################################### [100%]
Preparing...                ########################################### [100%]
   1:testutf8               ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-packagemanager   ########################################### [100%]
Updating etc/sysconfig/onlineupdate...
Preparing...                ########################################### [100%]
   1:yast2-core             ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-qt               ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-theme-Linux      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:perl-Config-Crontab    ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2                  ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-ncurses          ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-control-center   ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-x11              ########################################### [100%]
Preparing...                ########################################### [100%]
   1:perl-gettext           ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-perl-bindings    ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-country          ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-mail-aliases     ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-mouse            ########################################### [100%]
Preparing...                ########################################### [100%]
   1:ldapcpplib             ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-ldap             ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-online-update    ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-xml              ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-transfer         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:autoyast2-installation ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-packager         ########################################### [  9%]
   2:yast2-pam              ########################################### [ 18%]
   3:yast2-storage          ########################################### [ 27%]
   4:yast2-bootloader       ########################################### [ 36%]
Updating etc/sysconfig/bootloader...
   5:yast2-security         ########################################### [ 45%]
   6:yast2-network          ########################################### [ 55%]
   7:yast2-inetd            ########################################### [ 64%]
   8:yast2-ldap-client      ########################################### [ 73%]
Updating etc/sysconfig/ldap...
   9:yast2-users            ########################################### [ 82%]
  10:yast2-installation     ########################################### [ 91%]
  11:yast2-update           ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-sysconfig        ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-tune             ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-powertweak       ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-runlevel         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-sound            ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-bg         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-bs         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-cs         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-cy         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-da         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-de         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-el_GR      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-en_GB      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-en_US      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-es         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-fi         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-fr         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-hu         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-it         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-ja         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-ko         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-lt         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-nb         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-nl         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-pl         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-pt_BR      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-ro         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-ru         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-sk         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-sl_SI      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-sv         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-tr         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-zh_CN      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-trans-zh_TW      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-nis-client       ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-firewall         ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-http-server      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-mail             ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-kerberos-client  ########################################### [100%]
Preparing...                ########################################### [100%]
   1:perl-Parse-RecDescent  ########################################### [100%]
Preparing...                ########################################### [100%]
   1:perl-X500-DN           ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-ldap-server      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-nfs-client       ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-nfs-server       ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-ntp-client       ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-samba-client     ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-printer          ########################################### [100%]
Preparing...                ########################################### [100%]
   1:perl-Digest-MD4        ########################################### [100%]
Preparing...                ########################################### [100%]
   1:perl-Crypt-SmbHash     ########################################### [100%]
Preparing...                ########################################### [100%]
   1:nmbstatus              ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-samba-server     ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-tftp-server      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-support          ########################################### [100%]
Preparing...                ########################################### [100%]
   1:xmset                  ########################################### [100%]
Preparing...                ########################################### [100%]
   1:powertweak             ########################################### [100%]
Preparing...                ########################################### [100%]
   1:powertweak-extra       ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-dns-server       ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-dhcp-server      ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast2-nis-server       ########################################### [100%]
Preparing...                ########################################### [100%]
   1:yast-EM                ########################################### [100%]
Finished. Use /sbin/yast or /sbin/yast2 to start it

.
4) Once installed Yast could be invoked in three ways:

A terminal like interface

A cool GUI interface

On Enterprise Manager Grid Control go to “Targets -> Hosts -> Administration”

↧

Diagnosing and Resolving “gc block lost”

June 7, 2009, 12:45 am

≫ Next: howto: github – talks branch

≪ Previous: Yast on OEL

Last week, one of our clients had a sudden slow down on all of their applications which is running on two node RAC environment

Below is the summary of the setup:
– Server and Storage: SunFire X4200 with LUNs on EMC CX300
– OS: RHEL 4.3 ES
– Oracle 10.2.0.3 (database and clusterware)
– Database Files, Flash Recovery Area, OCR, and Voting disk are located on OCFS2 filesystems
– Application: Forms and Reports (6i and also lower)

As per the DBA, the workload on the database was normal and there were no changes on the RAC nodes and on the applications. Hmm, I can’t really tell because I haven’t really looked into their workload so I don’t have past data to compare.

I first setup the OS Watcher on both nodes (with 60sec snapshot interval) to have an end-to-end view of the server performance while I’m doing the diagnosis on the database. It’s a cool tool and automatically gathers a lot of info when you start it (cpu, io, memory, network, process info), also this tool has a graphing facility which makes it faster to detect spikes and any trends.

Then, I looked into the database parameters and compared it with the RDA of the database I had before. This is just to check if there are any changes on the parameters that might cause the sudden slow down. Hmm…there was nothing new

Then, I queried on the V$SESSION

SQL> select count() from v$session; -- server1* COUNT() ---------- 385 SQL> select count() from v$session; -- server2 COUNT(*) ---------- 49

.
I was surprised! The session count on both servers show that there are more users connected on server1 (almost 89% of the total users). This could be because of
1) the clients having lower versions (< Sql*Plus 8.1 or OCI8, see Note 97926.1) that may not support TAF (FAILOVER_MODE) and Load Balancing (LOAD_BALANCE) or
2) they are using TNS entries explicitly connecting to server1
Since the RAC database has different applications connected to it, knowing which applications are on server1 and how many they are will be enough to determine which users on particular applications could be transferred to server2 to at least balance the number of users on both nodes, but doing this may still not solve the issue as I don’t have enough info that this is the root cause of the problem..

The query below shows that all the users don’t have FAILOVER capabilities which could be attributed by the two points I mentioned above

select distinct s.inst_id instance_id, s.failover_type failover_type, s.failover_method
failover_method, s.failed_over failed_over
from gv$session s;
INSTANCE_ID FAILOVER_TYPE        FAILOVER_METHOD      FAILED_OVER
----------- -------------------- -------------------- --------------------
          1 NONE                 NONE                 NO
          2 NONE                 NONE                 NO

.
Below query shows the distinct applications/modules and their count which are all on server1. Also you’ll notice that most of them are inactive (some of these users should connect to server2 to have balance on both nodes)

SQL> select distinct inst_id, module, program, status, count(*)
from gv$session
group by inst_id, module, program, status
order by 1,2,3,4;

   INST_ID MODULE					      PROGRAM						 STATUS       COUNT(*)
---------- -------------------------------------------------- -------------------------------------------------- ---------- ----------
	 1 F45RUN32.EXE 				      F45RUN32.EXE					 INACTIVE	    23		 

... output snipped ....

	 1 xxxPgm					      xxxPgm						 INACTIVE	    21		

... output snipped ....

	 1 c:\xxxwin95\BIN\F45RUN32.EXE L:\xxx\VS\FORMS\MAI   c:\xxxwin95\BIN\F45RUN32.EXE L:\xxx\VS\FORMS\MAI	 INACTIVE	    44
	 1 c:\xxxwin95\BIN\F45RUN32.EXE L:\MAIN.FMX	      c:\xxxwin95\BIN\F45RUN32.EXE L:\MAIN.FMX		 INACTIVE	    31
	 1 c:\xxxwin95\BIN\F45RUN32.EXE w:\xxxMENU.fmx       c:\xxxwin95\BIN\F45RUN32.EXE w:\xxxMENU.fmx	 INACTIVE	    14
	 1 c:\xxxwin95\BIN\F45RUN32.EXE w:\xxxlogin.fmx       c:\xxxwin95\BIN\F45RUN32.EXE w:\xxxlogin.fmx	 INACTIVE	    96	

... output snipped ....

	 1 c:\xxxwin95\BIN\R25SRV32.exe 		      c:\xxxwin95\BIN\R25SRV32.exe			 INACTIVE	    54		

... output snipped ....

	 1 ifweb90.exe					      ifweb90.exe					 INACTIVE	    11

.
Below info shows some data samples of DBA_HIST_ACTIVE_SESS_HISTORY which is graphed using Tanel Poder’s Perfsheet (could be found here and here) to clearly distinguish the distribution of the modules and number of users. Most of the users on server1 uses the “c:\xxxwin95\BIN\F45RUN32.EXE w:\xxxlogin.fmx”

Having the graph above may lead us to conclusion that the significant number of users on server1 attributes to the slow down of the transactions. But as per the DBA, there were no changes made and they were running the same transactions as before on server1 which has acceptable response time. Also the OS Watcher shows that the CPU utilization (peak is 60% and most of the time at 40%) and run queue was low and the disk IO utilization and service time were average (always 10ms below), and there were enough memory on the server (3GB free out of 12GB.. the SGA is 2GB) and no swapping.

Hmm…Drilling down on the wait interface and on per session level that is doing the important business operations will give us a definite conclusion on what is really the bottleneck on the database.

The graph below is another sample from DBA_HIST_ACTIVE_SESS_HISTORY that shows server1 (in blue box) is suffering from “gc cr block lost” and “gc cr multi block request” from 7am to 4pm. The “Metalink Doc ID: 563566.1 gc lost blocks diagnostics” indicates that it is a cluster problem which could be a problem on the network interconnect

To confirm the bottleneck shown from the sample of DBA_HIST_ACTIVE_SESS_HISTORY (above), the output of ADDM and AWR report from a peak period was analyzed (from 2-3 & 3-4 pm). Below are the output of ADDM and AWR reports:

ADDM output:

          DETAILED ADDM REPORT FOR TASK 'TASK_137854' WITH ID 137854
          ----------------------------------------------------------

              Analysis Period: 28-MAY-2009 from 13:58:59 to 14:59:12
         Database ID/Instance: 3967623528/2
      Database/Instance Names: xxx
                    Host Name: xxx
             Database Version: 10.2.0.3.0
               Snapshot Range: from 15642 to 15643
                Database Time: 3710 seconds
        Average Database Load: 1 active sessions

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

FINDING 1: 52% impact (1917 seconds)
------------------------------------
Cluster multi-block requests were consuming significant database time.

   RECOMMENDATION 1: SQL Tuning, 37% benefit (1389 seconds)
      ACTION: Run SQL Tuning Advisor on the SQL statement with SQL_ID
         "0h0fn2d19adtk". Look for an alternative plan that does not use
         object scans.
         RELEVANT OBJECT: SQL statement with SQL_ID 0h0fn2d19adtk

         ... output snipped ... 

   RECOMMENDATION 2: SQL Tuning, 8.4% benefit (310 seconds)
      ACTION: Run SQL Tuning Advisor on the SQL statement with SQL_ID
         "8jd43xr4rp00u". Look for an alternative plan that does not use
         object scans.
         RELEVANT OBJECT: SQL statement with SQL_ID 8jd43xr4rp00u

         ... output snipped ... 

   RECOMMENDATION 3: SQL Tuning, 4.6% benefit (172 seconds)
      ACTION: Run SQL Tuning Advisor on the SQL statement with SQL_ID
         "6wpgf3s0vzuks". Look for an alternative plan that does not use
         object scans.
         RELEVANT OBJECT: SQL statement with SQL_ID 6wpgf3s0vzuks

         ... output snipped ... 

   SYMPTOMS THAT LED TO THE FINDING:
      SYMPTOM: Inter-instance messaging was consuming significant database
               time on this instance. (94% impact [3488 seconds])
         SYMPTOM: Wait class "Cluster" was consuming significant database
                  time. (94% impact [3488 seconds])

FINDING 2: 51% impact (1874 seconds)
------------------------------------
Global Cache Service Processes (LMSn) in other instances were not processing
requests fast enough.

   RECOMMENDATION 1: DB Configuration, 51% benefit (1874 seconds)
      ACTION: Increase throughput of the Global Cache Service (LMSn)
         processes. Increase the number of Global Cache Service processes by
         increasing the value of the parameter "gcs_server_processes".
         Alternatively, if the host is CPU bound consider increasing the OS
         priority of the Global Cache Service processes.
      RATIONALE: The value of parameter "gcs_server_processes" was "2" during
         the analysis period.

   SYMPTOMS THAT LED TO THE FINDING:
      SYMPTOM: Inter-instance messaging was consuming significant database
               time on this instance. (94% impact [3488 seconds])
         SYMPTOM: Wait class "Cluster" was consuming significant database
                  time. (94% impact [3488 seconds])

FINDING 3: 36% impact (1330 seconds)
------------------------------------
Cluster communications that were retried due to lost blocks consumed
significant database time.

   RECOMMENDATION 1: Host Configuration, 36% benefit (1330 seconds)
      ACTION: Check the configuration of the cluster interconnect. Check OS
         setup like adapter setting, firmware and driver release. Check that
         the OS's socket receive buffers are large enough to store an entire
         multiblock read. The value of parameter
         "db_file_multiblock_read_count" may be decreased as a workaround.
      RATIONALE: The instance was consuming 477 kilo bits per second of
         interconnect bandwidth.

   SYMPTOMS THAT LED TO THE FINDING:
      SYMPTOM: Inter-instance messaging was consuming significant database
               time on this instance. (94% impact [3488 seconds])
         SYMPTOM: Wait class "Cluster" was consuming significant database
                  time. (94% impact [3488 seconds])

#######################################################################

          DETAILED ADDM REPORT FOR TASK 'TASK_137874' WITH ID 137874
          ----------------------------------------------------------

              Analysis Period: 28-MAY-2009 from 14:59:12 to 15:58:44
         Database ID/Instance: 3967623528/2
      Database/Instance Names: xxx
                    Host Name: xxx
             Database Version: 10.2.0.3.0
               Snapshot Range: from 15643 to 15644
                Database Time: 1563 seconds
        Average Database Load: .4 active sessions

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

FINDING 1: 42% impact (654 seconds)
-----------------------------------
Cluster communications that were retried due to lost blocks consumed
significant database time.

   RECOMMENDATION 1: Host Configuration, 42% benefit (654 seconds)
      ACTION: Check the configuration of the cluster interconnect. Check OS
         setup like adapter setting, firmware and driver release. Check that
         the OS's socket receive buffers are large enough to store an entire
         multiblock read. The value of parameter
         "db_file_multiblock_read_count" may be decreased as a workaround.
      RATIONALE: The instance was consuming 134 kilo bits per second of
         interconnect bandwidth.

   SYMPTOMS THAT LED TO THE FINDING:
      SYMPTOM: Inter-instance messaging was consuming significant database
               time on this instance. (88% impact [1371 seconds])
         SYMPTOM: Wait class "Cluster" was consuming significant database
                  time. (88% impact [1372 seconds])

.
AWR output:

WORKLOAD REPOSITORY report for

DB Name         DB Id    Instance     Inst Num Release     RAC Host
------------ ----------- ------------ -------- ----------- --- ------------
xxxxx         3967623528 xxxxxx              2 10.2.0.3.0  YES xxx

              Snap Id      Snap Time      Sessions Curs/Sess
            --------- ------------------- -------- ---------
Begin Snap:     15642 28-May-09 13:58:58        57      27.0
  End Snap:     15643 28-May-09 14:59:12        64      28.7
   Elapsed:               60.23 (mins)
   DB Time:               61.83 (mins)

Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
gc cr multi block request            19,644       1,902     97   51.3    Cluster
gc cr block lost                      1,011       1,117   1104   30.1    Cluster
enq: WF - contention                    626         268    429    7.2      Other
gc current block lost                   186         203   1089    5.5    Cluster
cr request retry                        678         162    240    4.4      Other

Per Wait Class
~~~~~~~~~~~~~~~~~~
                                                                  Avg
                                       %Time       Total Wait    wait     Waits
Wait Class                      Waits  -outs         Time (s)    (ms)      /txn
-------------------- ---------------- ------ ---------------- ------- ---------
Cluster                        34,450    7.0            3,385      98       6.8
Other                          91,292   64.7              467       5      18.1
User I/O                       26,510     .0              102       4       5.3
Concurrency                    29,668     .0               54       2       5.9
System I/O                     13,360     .0               11       1       2.6
Commit                          1,313     .0                4       3       0.3
Application                     2,374     .0                2       1       0.5
Network                       129,774     .0                0       0      25.7
Configuration                      20     .0                0       0       0.0

#######################################################################

WORKLOAD REPOSITORY report for

DB Name         DB Id    Instance     Inst Num Release     RAC Host
------------ ----------- ------------ -------- ----------- --- ------------
xxxxx         3967623528 xxxxxx              2 10.2.0.3.0  YES xxx

              Snap Id      Snap Time      Sessions Curs/Sess
            --------- ------------------- -------- ---------
Begin Snap:     15643 28-May-09 14:59:12        64      28.7
  End Snap:     15644 28-May-09 15:58:43        64      29.6
   Elapsed:               59.53 (mins)
   DB Time:               26.05 (mins)

Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
gc cr block lost                        389         429   1103   27.5    Cluster
gc cr multi block request             2,360         422    179   27.0    Cluster
enq: WF - contention                    510         212    416   13.6      Other
gc current block lost                   188         204   1084   13.0    Cluster
cr request retry                        326         181    554   11.6      Other

Per Wait Class
~~~~~~~~~~~~~~~~~~
                                                                  Avg
                                       %Time       Total Wait    wait     Waits
Wait Class                      Waits  -outs         Time (s)    (ms)      /txn
-------------------- ---------------- ------ ---------------- ------- ---------
Cluster                        10,239    8.9            1,247     122       2.1
Other                         153,082   71.9              468       3      31.0
Application                     1,784   18.0              161      90       0.4
Concurrency                    33,014     .0               53       2       6.7
User I/O                        4,246     .0               27       6       0.9
System I/O                     13,648     .0               14       1       2.8
Commit                          1,546     .0                4       3       0.3
Network                       138,333     .0                1       0      28.0
Configuration                      16     .0                0       4       0.0

.
The only thing that is left to do is to drill down on the complaining users and check on the network performance and interconnect… unfortunately the users were going home and I can’t see any significant database activity. So I called it a night, and just analyzed my activity log and read on Metalink Doc ID 563566.1. Also I advised the DBA to divide the users across the nodes.

Next day afternoon, and it’s time to do the drill down. You can see on the graph below (a sample from GV$SESSION) that the users were already distributed, but still there is a performance problem. I told you this will not solve the issue ! :) it is just a plus that we manage to distribute the users :)

A query on GV$SESSION_LONGOPS will show users with large SQLs. Drilling down on the session statistics of these user’s will give detail on what are they waiting and how much time they spent on what resource. However, it would be a repetitive step to get all the sessions on the GV$SESSION_LONGOPS.

select * from (
			select
			a.inst_id inst, a.START_TIME, b.osuser, b.module,  a.sid, a.serial#,
			a.opname, b.event, a.target, a.sofar, a.totalwork, round(a.sofar/a.totalwork, 4)*100 pct, a.units, a.elapsed_seconds, a.time_remaining time_remaining_sec, round(a.time_remaining/60,2) min
			,a.sql_hash_value
		-- 	,message
			from gv$session_longops a, gv$session b
			where a.sid = b.sid
			and b.serial# = b.serial#
			and sofar < totalwork
			order by totalwork desc)

INST START_TIME        OSUSER     MODULE             SID SERIAL# OPNAME     EVENT           TARGET          SOFAR  TOTALWORK        PCT UNITS      ELAPSED_SECONDS TIME_REMAINING_SEC        MIN SQL_HASH_VALUE
---- ----------------- ---------- --------------- ------ ------- ---------- --------------- ---------- ---------- ---------- ---------- ---------- --------------- ------------------ ---------- --------------
   2 20090529 18:36:38 oraprod    oracle@xxxxx280    454      82 Table Scan gc cr multi blo xxxxxxxx.B       2205      25403       8.68 Blocks                 343               3609      60.15     1857468103
                                  R (TNS V1-V3)                             ck request      ENMAS

   2 20090529 18:30:39 oraprod    oracle@xxxxx280    607     546 Table Scan gc cr multi blo xxxxxxxx.C      11062      15203      72.76 Blocks                 708                265       4.42     1477847824
                                  R (TNS V1-V3)                             ck request      OVMAS

   1 20090529 15:51:58 PC829      c:\xxxwin95\BIN    474     721 Table Scan gc cr request   xxxxxxxx.C      11108      15203      73.06 Blocks               10317               3803      63.38     1636678807
                                  \F45RUN32.EXE w                                           OVMAS
                                  : \xxxlogin.fmx

   1 20090529 17:27:49 PC1280     c:\xxxwin95\BIN    473     171 Table Scan gc cr request   xxxxxx.DTT        336      12918        2.6 Blocks                4449             166599    2776.65     1117075250
                                  \R25SRV32.exe                                             _DVDTL

   1 20090529 17:27:49 oracle     sqlplus@server2    473     171 Table Scan PX Deq: Executi xxxxxx.DTT        336      12918        2.6 Blocks                4449             166599    2776.65     1117075250
                                  .xxxx.xxx (TNS                           on Msg          _DVDTL
                                   V1-V3)

.
I already have an idea that my problem is on the cluster configuration. So I asked the DBA that we go up to the Accounting Department (where there are a lot of users complaining) and check on the desktop terminals of the users. We saw two users which has the application module on total hang. These users are PC1069 (with SID 601) and PC918 (with SID 483). So a query on the GV$SESSION_LONGOPS does not show these users but a query on GV$SESSION_WAIT shows that they are waiting on “gc cr request” event

select s.inst_id, w.sid, s.sql_hash_value, s.osuser, s.program, w.event, w.seconds_in_wait seconds_in_wait, w.state, w.p1, w.p2, w.p3
from gv$session s, gv$session_wait w
where s.sid = w.sid and s.type = 'USER'
and w.state = 'WAITING'
and s.osuser in ('PC1069','PC918')
order by 6 asc;

   INST_ID    SID SQL_HASH_VALUE OSUSER     PROGRAM                 EVENT              SECONDS_IN_WAIT STATE                     P1         P2         P3
---------- ------ -------------- ---------- ----------------------- ------------------ --------------- ----------------- ---------- ---------- ----------
         1    483      449494677 PC918      c:\xxxwin95\BIN\F45RUN3 SQL*Net message fr            3853 WAITING            675562835          1          0
                                            2.EXE w:\xxxlogin.fmx   om client

         1    601     1874068660 PC1069     c:\xxxwin95\BIN\F45RUN3 gc cr request                    6 WAITING                   12    1173863          1
                                            2.EXE w:\xxxlogin.fmx

         1    483      449494677 PC918      c:\xxxwin95\BIN\F45RUN3 gc cr request                    0 WAITING                    7    1649989          1
                                            2.EXE w:\xxxlogin.fmx

.
Focusing on these two sessions (SID 601 and 483) and getting their session statistics will give us an idea if they are doing something, on a total hang, or waiting for something. Below query on V$SESSION_WAIT shows that the SEQ# is being incremented (a number that uniquely identifies this wait) which means the session (SID 601) is still waiting for something, the event “gc cr request”

BTW, I’m using SW script of Tanel Poder (downloadable here), and you’ll also see some usage of Snapper and USQL

SQL> @sw 601

    SID STATE   EVENT                                          SEQ# SEC_IN_WAIT         P1         P2         P3 P1TRANSL
------- ------- ---------------------------------------- ---------- ----------- ---------- ---------- ---------- ------------------------------------------
    601 WAITING gc cr request                                 27223           3          7    1647259          1

... output snipped ...

SQL> @sw 601

    SID STATE   EVENT                                          SEQ# SEC_IN_WAIT         P1         P2         P3 P1TRANSL
------- ------- ---------------------------------------- ---------- ----------- ---------- ---------- ---------- ------------------------------------------
    601 WAITING gc cr request                                 27225           0          7    1647259          1

... output snipped ...

SQL> @sw 601

    SID STATE   EVENT                                          SEQ# SEC_IN_WAIT         P1         P2         P3 P1TRANSL
------- ------- ---------------------------------------- ---------- ----------- ---------- ---------- ---------- ------------------------------------------
    601 WAITING gc cr request                                 27227           0          7    1647259          1

... output snipped ...

SQL> @sw 601

    SID STATE   EVENT                                          SEQ# SEC_IN_WAIT         P1         P2         P3 P1TRANSL
------- ------- ---------------------------------------- ---------- ----------- ---------- ---------- ---------- ------------------------------------------
    601 WAITING gc cr request                                 27229           7          7    1647259          1

.
A query on V$SESSTAT will show the performance/wait counters being incremented (SID 601), the output below was a snap of the session for 5 seconds period done 5 times. You’ll see that “gc current” waits are significant

SQL> @snapper out 5 5 601
-- Session Snapper v1.10 by Tanel Poder ( http://www.tanelpoder.com )
---------------------------------------------------------------------------------------------------------------------------------------------
HEAD,     SID, SNAPSHOT START   ,  SECONDS, TYPE, STATISTIC                               ,         DELTA,  DELTA/SEC,     HDELTA, HDELTA/SEC
---------------------------------------------------------------------------------------------------------------------------------------------
DATA,     601, 20090529 18:48:36,        6, STAT, opened cursors current                  ,            -1,         -0,         -1,       -.17
DATA,     601, 20090529 18:48:36,        6, STAT, recursive calls                         ,             5,          1,          5,        .83
DATA,     601, 20090529 18:48:36,        6, STAT, session logical reads                   ,            24,          4,         24,          4
DATA,     601, 20090529 18:48:36,        6, STAT, cluster wait time                       ,           492,         82,        492,         82
DATA,     601, 20090529 18:48:36,        6, STAT, user I/O wait time                      ,             1,          0,          1,        .17
DATA,     601, 20090529 18:48:36,        6, STAT, session pga memory                      ,        -65536,     -10923,    -65.54k,    -10.92k
DATA,     601, 20090529 18:48:36,        6, STAT, physical read total IO requests         ,             1,          0,          1,        .17
DATA,     601, 20090529 18:48:36,        6, STAT, physical read total bytes               ,          8192,       1365,      8.19k,      1.37k
DATA,     601, 20090529 18:48:36,        6, STAT, gcs messages sent                       ,             3,          1,          3,         .5
DATA,     601, 20090529 18:48:36,        6, STAT, db block gets                           ,             6,          1,          6,          1
DATA,     601, 20090529 18:48:36,        6, STAT, db block gets from cache                ,             6,          1,          6,          1
DATA,     601, 20090529 18:48:36,        6, STAT, consistent gets                         ,            18,          3,         18,          3
DATA,     601, 20090529 18:48:36,        6, STAT, consistent gets from cache              ,            18,          3,         18,          3
DATA,     601, 20090529 18:48:36,        6, STAT, consistent gets - examination           ,             8,          1,          8,       1.33
DATA,     601, 20090529 18:48:36,        6, STAT, physical reads                          ,             1,          0,          1,        .17
DATA,     601, 20090529 18:48:36,        6, STAT, physical reads cache                    ,             1,          0,          1,        .17
DATA,     601, 20090529 18:48:36,        6, STAT, physical read IO requests               ,             1,          0,          1,        .17
DATA,     601, 20090529 18:48:36,        6, STAT, physical read bytes                     ,          8192,       1365,      8.19k,      1.37k
DATA,     601, 20090529 18:48:36,        6, STAT, db block changes                        ,             3,          1,          3,         .5
DATA,     601, 20090529 18:48:36,        6, STAT, free buffer requested                   ,             3,          1,          3,         .5
DATA,     601, 20090529 18:48:36,        6, STAT, calls to get snapshot scn: kcmgss       ,             5,          1,          5,        .83
DATA,     601, 20090529 18:48:36,        6, STAT, redo entries                            ,             2,          0,          2,        .33
DATA,     601, 20090529 18:48:36,        6, STAT, redo size                               ,           536,         89,        536,      89.33
DATA,     601, 20090529 18:48:36,        6, STAT, gc current blocks received              ,             1,          0,          1,        .17
DATA,     601, 20090529 18:48:36,        6, STAT, gc remote grants                        ,             1,          0,          1,        .17
DATA,     601, 20090529 18:48:36,        6, STAT, undo change vector size                 ,            84,         14,         84,         14
DATA,     601, 20090529 18:48:36,        6, STAT, no work - consistent read gets          ,             9,          2,          9,        1.5
DATA,     601, 20090529 18:48:36,        6, STAT, table fetch by rowid                    ,             4,          1,          4,        .67
DATA,     601, 20090529 18:48:36,        6, STAT, index scans kdiixs1                     ,             4,          1,          4,        .67
DATA,     601, 20090529 18:48:36,        6, STAT, buffer is not pinned count              ,             8,          1,          8,       1.33
DATA,     601, 20090529 18:48:36,        6, STAT, execute count                           ,             4,          1,          4,        .67
DATA,     601, 20090529 18:48:36,        6, TIME, PL/SQL execution elapsed time           ,           139,         23,      139us,    23.17us
DATA,     601, 20090529 18:48:36,        6, TIME, DB CPU                                  ,         10000,       1667,       10ms,     1.67ms
DATA,     601, 20090529 18:48:36,        6, TIME, sql execute elapsed time                ,      27552125,    4592021,     27.55s,      4.59s
DATA,     601, 20090529 18:48:36,        6, TIME, DB time                                 ,      27552133,    4592022,     27.55s,      4.59s
DATA,     601, 20090529 18:48:36,        6, WAIT, db file sequential read                 ,          8374,       1396,     8.37ms,      1.4ms
DATA,     601, 20090529 18:48:36,        6, WAIT, gc current request                      ,       9664835,    1610806,      9.66s,      1.61s
--  End of snap 1
DATA,     601, 20090529 18:48:42,        5, STAT, opened cursors cumulative               ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, opened cursors current                  ,            -1,         -0,         -1,        -.2
DATA,     601, 20090529 18:48:42,        5, STAT, recursive calls                         ,             4,          1,          4,         .8
DATA,     601, 20090529 18:48:42,        5, STAT, recursive cpu usage                     ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, session logical reads                   ,            42,          8,         42,        8.4
DATA,     601, 20090529 18:48:42,        5, STAT, cluster wait time                       ,           419,         84,        419,       83.8
DATA,     601, 20090529 18:48:42,        5, STAT, user I/O wait time                      ,             2,          0,          2,         .4
DATA,     601, 20090529 18:48:42,        5, STAT, enqueue requests                        ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, global enqueue gets sync                ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, physical read total IO requests         ,             2,          0,          2,         .4
DATA,     601, 20090529 18:48:42,        5, STAT, physical read total bytes               ,         16384,       3277,     16.38k,      3.28k
DATA,     601, 20090529 18:48:42,        5, STAT, gcs messages sent                       ,             2,          0,          2,         .4
DATA,     601, 20090529 18:48:42,        5, STAT, db block gets                           ,            41,          8,         41,        8.2
DATA,     601, 20090529 18:48:42,        5, STAT, db block gets from cache                ,            41,          8,         41,        8.2
DATA,     601, 20090529 18:48:42,        5, STAT, consistent gets                         ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, consistent gets from cache              ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, physical reads                          ,             2,          0,          2,         .4
DATA,     601, 20090529 18:48:42,        5, STAT, physical reads cache                    ,             2,          0,          2,         .4
DATA,     601, 20090529 18:48:42,        5, STAT, physical read IO requests               ,             2,          0,          2,         .4
DATA,     601, 20090529 18:48:42,        5, STAT, physical read bytes                     ,         16384,       3277,     16.38k,      3.28k
DATA,     601, 20090529 18:48:42,        5, STAT, db block changes                        ,            26,          5,         26,        5.2
DATA,     601, 20090529 18:48:42,        5, STAT, exchange deadlocks                      ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, free buffer requested                   ,             3,          1,          3,         .6
DATA,     601, 20090529 18:48:42,        5, STAT, calls to get snapshot scn: kcmgss       ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, redo entries                            ,            14,          3,         14,        2.8
DATA,     601, 20090529 18:48:42,        5, STAT, redo size                               ,          3504,        701,       3.5k,      700.8
DATA,     601, 20090529 18:48:42,        5, STAT, redo subscn max counts                  ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, gc local grants                         ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, gc remote grants                        ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, undo change vector size                 ,          1140,        228,      1.14k,        228
DATA,     601, 20090529 18:48:42,        5, STAT, deferred (CURRENT) block cleanout applic,             4,          1,          4,         .8
DATA,     601, 20090529 18:48:42,        5, STAT, parse count (total)                     ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, STAT, execute count                           ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:42,        5, TIME, parse time elapsed                      ,            41,          8,       41us,      8.2us
DATA,     601, 20090529 18:48:42,        5, TIME, PL/SQL execution elapsed time           ,           125,         25,      125us,       25us
DATA,     601, 20090529 18:48:42,        5, TIME, sql execute elapsed time                ,       5433501,    1086700,      5.43s,      1.09s
DATA,     601, 20090529 18:48:42,        5, TIME, DB time                                 ,       5433501,    1086700,      5.43s,      1.09s
DATA,     601, 20090529 18:48:42,        5, WAIT, db file sequential read                 ,         17812,       3562,    17.81ms,     3.56ms
DATA,     601, 20090529 18:48:42,        5, WAIT, gc current request                      ,      -4221489,    -844298,     -4.22s,   -844.3ms
DATA,     601, 20090529 18:48:42,        5, WAIT, gc current block lost                   ,       6634963,    1326993,      6.63s,      1.33s
DATA,     601, 20090529 18:48:42,        5, WAIT, gc current grant 2-way                  ,           254,         51,      254us,     50.8us
DATA,     601, 20090529 18:48:42,        5, WAIT, events in waitclass Other               ,           600,        120,      600us,      120us
--  End of snap 2
DATA,     601, 20090529 18:48:47,        5, STAT, session logical reads                   ,             3,          1,          3,         .6
DATA,     601, 20090529 18:48:47,        5, STAT, cluster wait time                       ,           503,        101,        503,      100.6
DATA,     601, 20090529 18:48:47,        5, STAT, gcs messages sent                       ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:47,        5, STAT, db block gets                           ,             3,          1,          3,         .6
DATA,     601, 20090529 18:48:47,        5, STAT, db block gets from cache                ,             3,          1,          3,         .6
DATA,     601, 20090529 18:48:47,        5, STAT, exchange deadlocks                      ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:47,        5, STAT, free buffer requested                   ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:47,        5, WAIT, gc current request                      ,       1221742,     244348,      1.22s,   244.35ms
DATA,     601, 20090529 18:48:47,        5, WAIT, gc current block lost                   ,       3813435,     762687,      3.81s,   762.69ms
DATA,     601, 20090529 18:48:47,        5, WAIT, events in waitclass Other               ,           609,        122,      609us,    121.8us
--  End of snap 3
DATA,     601, 20090529 18:48:52,        5, STAT, session logical reads                   ,             3,          1,          3,         .6
DATA,     601, 20090529 18:48:52,        5, STAT, cluster wait time                       ,           524,        105,        524,      104.8
DATA,     601, 20090529 18:48:52,        5, STAT, gcs messages sent                       ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:52,        5, STAT, db block gets                           ,             3,          1,          3,         .6
DATA,     601, 20090529 18:48:52,        5, STAT, db block gets from cache                ,             3,          1,          3,         .6
DATA,     601, 20090529 18:48:52,        5, STAT, exchange deadlocks                      ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:52,        5, STAT, free buffer requested                   ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:52,        5, TIME, sql execute elapsed time                ,       6613095,    1322619,      6.61s,      1.32s
DATA,     601, 20090529 18:48:52,        5, TIME, DB time                                 ,       6613095,    1322619,      6.61s,      1.32s
DATA,     601, 20090529 18:48:52,        5, WAIT, gc current request                      ,       3000843,     600169,         3s,   600.17ms
DATA,     601, 20090529 18:48:52,        5, WAIT, gc current block lost                   ,       4018577,     803715,      4.02s,   803.72ms
DATA,     601, 20090529 18:48:52,        5, WAIT, events in waitclass Other               ,           570,        114,      570us,      114us
--  End of snap 4
DATA,     601, 20090529 18:48:57,        5, STAT, session logical reads                   ,             3,          1,          3,         .6
DATA,     601, 20090529 18:48:57,        5, STAT, cluster wait time                       ,           523,        105,        523,      104.6
DATA,     601, 20090529 18:48:57,        5, STAT, gcs messages sent                       ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:57,        5, STAT, db block gets                           ,             3,          1,          3,         .6
DATA,     601, 20090529 18:48:57,        5, STAT, db block gets from cache                ,             3,          1,          3,         .6
DATA,     601, 20090529 18:48:57,        5, STAT, exchange deadlocks                      ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:57,        5, STAT, free buffer requested                   ,             1,          0,          1,         .2
DATA,     601, 20090529 18:48:57,        5, TIME, sql execute elapsed time                ,       5230170,    1046034,      5.23s,      1.05s
DATA,     601, 20090529 18:48:57,        5, TIME, DB time                                 ,       5230170,    1046034,      5.23s,      1.05s
DATA,     601, 20090529 18:48:57,        5, WAIT, gc current request                      ,          -473,        -95,     -473us,    -94.6us
DATA,     601, 20090529 18:48:57,        5, WAIT, gc current block lost                   ,       5230137,    1046027,      5.23s,      1.05s
DATA,     601, 20090529 18:48:57,        5, WAIT, events in waitclass Other               ,           390,         78,      390us,       78us
--  End of snap 5

PL/SQL procedure successfully completed.

.
Below is the SQL being executed for SID 601 which is a batch INSERT operation that is just a normal operation as per the DBA

SQL> @usql 601

HASH_VALUE  CH# SQL_TEXT
---------- ---- ------------------------------------------------------------------------------------------------------------------------------------------------------
   6143262    0 INSERT INTO xxx(
                BATxxx,POLxxx,SRCxxx,PAYxxx,DATxxx,DATxxx,AMxxxx,ACCxxx,NEWxxx,POLxxx,FUNxxx,Uxxxxx,Txxxx,Cxxxx,PRxxxx,Exxxx,xxxxID,POxxxx,Nxxxx,xxxAMT,xxxPRM,xxxMAV
                ,COLxxxx,NOPxxx )  VALUES ( :b1,:b2,:b3,:b4,:b5,:b6,:b7,:b8,:b9,:b10,:b11,:b12,:b13,:b14,:b15,:b16,:b17,:b18,:b19,:b20,:b21,:b22,:b23,:b24  )

 CH# PARENT_HANDLE    OBJECT_HANDLE        PARSES   H_PARSES EXECUTIONS    FETCHES ROWS_PROCESSED       LIOS       PIOS      SORTS     CPU_MS     ELA_MS USERS_EXECUTING
---- ---------------- ---------------- ---------- ---------- ---------- ---------- -------------- ---------- ---------- ---------- ---------- ---------- ---------------
   0 00000000D0A5FC68 00000000D0A5FA40         10          1       2714          0           2714     300753      17731          0  12136.359  5899798.3               1

SQL hash value:         6143262    Cursor address:  00000000D0A5FC68    |   Statement first parsed at: 2009-05-29/13:01:42  |  20903 seconds ago

.
Below is the SQL being executed for SID 483 which is a SELECT operation that is just a normal operation as per the DBA

SQL> @usql 483

HASH_VALUE  CH# SQL_TEXT
---------- ---- ------------------------------------------------------------------------------------------------------------------------------------------------------
 449494677    0 SELECT x.xxxPxD,F.xxxNMx,B.xxxPxD,x.xxxNMx,D.PxLMxD,D.PxLFRQ,x.xxVxMT,x.xxVSTx,x.xxVxFF,x.xxVDxx,SxM(G.xxVMxN) MxDPRxM   FRxM RxPMxS x,BxNMxS B,xxxMxS
                x,PxLMxS D,xxVMxS x,xxxMxS F,xxVMxS G  WHxRx x.PxLNxM = NVL(:b1,'X')  xND x.RxLTYP = 'PxY'  xND (x.RxLPRx = 100  xR x.RxLPRx xS NxLL  ) xND x.xxxPxD =
                F.xxxPxD  xND B.PxLNxM (+)   = NVL(:b2,'X')  xND B.RxLTYP (+)   = 'xNS'  xND B.xxVNxM = 1  xND (B.RxLPxR = 100  xR B.RxLPxR xS NxLL  ) xND B.xxxPxD =
                x.xxxPxD (+)    xND D.PxLNxM = NVL(:b3,'X')  xND x.PxLNxM = NVL(:b4,'X')  xND x.xxVNxM = 1  xND G.PxLNxM = NVL(:b5,'X')  GRxxP BY
                x.xxxPxD,F.xxxNMx,B.xxxPxD,x.xxxNMx,D.PxLMxD,D.PxLFRQ,x.xxVxMT,x.xxVSTx,x.xxVxFF,x.xxVDxx

 CH# PARENT_HANDLE    OBJECT_HANDLE        PARSES   H_PARSES EXECUTIONS    FETCHES ROWS_PROCESSED       LIOS       PIOS      SORTS     CPU_MS     ELA_MS USERS_EXECUTING
---- ---------------- ---------------- ---------- ---------- ---------- ---------- -------------- ---------- ---------- ---------- ---------- ---------- ---------------
   0 00000000C2E54DD0 00000000C2E5E700          8          1       1047       2089           1043      58793       3905       2087   2676.616 11108807.7               1

.
A query on V$SESSTAT will show the performance/wait counters being incremented (SID 483), the output below was a snap of the session for 5 seconds period done 10 times. You’ll see that “gc cr” waits are significant

SQL> @snapper out 5 10 483
-- Session Snapper v1.10 by Tanel Poder ( http://www.tanelpoder.com )
---------------------------------------------------------------------------------------------------------------------------------------------
HEAD,     SID, SNAPSHOT START   ,  SECONDS, TYPE, STATISTIC                               ,         DELTA,  DELTA/SEC,     HDELTA, HDELTA/SEC
---------------------------------------------------------------------------------------------------------------------------------------------
DATA,     483, 20090529 19:00:17,        5, STAT, cluster wait time                       ,           527,        105,        527,      105.4
DATA,     483, 20090529 19:00:17,        5, STAT, gcs messages sent                       ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:17,        5, STAT, free buffer requested                   ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:17,        5, WAIT, gc cr request                           ,      -1220551,    -244110,     -1.22s,  -244.11ms
DATA,     483, 20090529 19:00:17,        5, WAIT, gc cr block lost                        ,       6485830,    1297166,      6.49s,       1.3s
DATA,     483, 20090529 19:00:17,        5, WAIT, events in waitclass Other               ,           530,        106,      530us,      106us
--  End of snap 1
DATA,     483, 20090529 19:00:22,        5, STAT, session logical reads                   ,             4,          1,          4,         .8
DATA,     483, 20090529 19:00:22,        5, STAT, cluster wait time                       ,           454,         91,        454,       90.8
DATA,     483, 20090529 19:00:22,        5, STAT, messages sent                           ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:22,        5, STAT, gcs messages sent                       ,             2,          0,          2,         .4
DATA,     483, 20090529 19:00:22,        5, STAT, consistent gets                         ,             4,          1,          4,         .8
DATA,     483, 20090529 19:00:22,        5, STAT, consistent gets from cache              ,             4,          1,          4,         .8
DATA,     483, 20090529 19:00:22,        5, STAT, consistent gets - examination           ,             3,          1,          3,         .6
DATA,     483, 20090529 19:00:22,        5, STAT, free buffer requested                   ,             2,          0,          2,         .4
DATA,     483, 20090529 19:00:22,        5, STAT, shared hash latch upgrades - no wait    ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:22,        5, STAT, gc current blocks received              ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:22,        5, STAT, no work - consistent read gets          ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:22,        5, STAT, table fetch by rowid                    ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:22,        5, STAT, index fetch by key                      ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:22,        5, STAT, buffer is not pinned count              ,             2,          0,          2,         .4
DATA,     483, 20090529 19:00:22,        5, WAIT, gc cr request                           ,      -8443706,   -1688741,     -8.44s,     -1.69s
DATA,     483, 20090529 19:00:22,        5, WAIT, gc cr block lost                        ,       6986370,    1397274,      6.99s,       1.4s
DATA,     483, 20090529 19:00:22,        5, WAIT, gc current block 2-way                  ,           596,        119,      596us,    119.2us
DATA,     483, 20090529 19:00:22,        5, WAIT, events in waitclass Other               ,           266,         53,      266us,     53.2us
--  End of snap 2
DATA,     483, 20090529 19:00:27,        5, STAT, cluster wait time                       ,           506,        101,        506,      101.2
DATA,     483, 20090529 19:00:27,        5, STAT, gcs messages sent                       ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:27,        5, STAT, free buffer requested                   ,             1,          0,          1,         .2
--  End of snap 3
DATA,     483, 20090529 19:00:32,        5, STAT, cluster wait time                       ,           488,         98,        488,       97.6
DATA,     483, 20090529 19:00:32,        5, WAIT, gc cr request                           ,       9888169,    1977634,      9.89s,      1.98s
--  End of snap 4
DATA,     483, 20090529 19:00:37,        5, STAT, cluster wait time                       ,           548,        110,        548,      109.6
DATA,     483, 20090529 19:00:37,        5, STAT, gcs messages sent                       ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:37,        5, STAT, free buffer requested                   ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:37,        5, WAIT, gc cr request                           ,      -4444128,    -888826,     -4.44s,  -888.83ms
DATA,     483, 20090529 19:00:37,        5, WAIT, gc cr block lost                        ,       7920880,    1584176,      7.92s,      1.58s
DATA,     483, 20090529 19:00:37,        5, WAIT, events in waitclass Other               ,           341,         68,      341us,     68.2us
--  End of snap 5
DATA,     483, 20090529 19:00:42,        5, STAT, cluster wait time                       ,           494,         99,        494,       98.8
DATA,     483, 20090529 19:00:42,        5, STAT, gcs messages sent                       ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:42,        5, STAT, free buffer requested                   ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:42,        5, WAIT, gc cr request                           ,        883591,     176718,   883.59ms,   176.72ms
DATA,     483, 20090529 19:00:42,        5, WAIT, gc cr block lost                        ,       4055130,     811026,      4.06s,   811.03ms
DATA,     483, 20090529 19:00:42,        5, WAIT, events in waitclass Other               ,           337,         67,      337us,     67.4us
--  End of snap 6
DATA,     483, 20090529 19:00:47,        6, STAT, cluster wait time                       ,           448,         75,        448,      74.67
DATA,     483, 20090529 19:00:47,        6, STAT, gcs messages sent                       ,             1,          0,          1,        .17
DATA,     483, 20090529 19:00:47,        6, STAT, free buffer requested                   ,             1,          0,          1,        .17
DATA,     483, 20090529 19:00:47,        6, WAIT, gc cr request                           ,      -2105578,    -350930,     -2.11s,  -350.93ms
DATA,     483, 20090529 19:00:47,        6, WAIT, gc cr block lost                        ,       6590281,    1098380,      6.59s,       1.1s
DATA,     483, 20090529 19:00:47,        6, WAIT, events in waitclass Other               ,           496,         83,      496us,    82.67us
--  End of snap 7
DATA,     483, 20090529 19:00:53,        5, STAT, cluster wait time                       ,           547,        109,        547,      109.4
DATA,     483, 20090529 19:00:53,        5, STAT, gcs messages sent                       ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:53,        5, STAT, free buffer requested                   ,             1,          0,          1,         .2
--  End of snap 8
DATA,     483, 20090529 19:00:58,        5, STAT, cluster wait time                       ,           528,        106,        528,      105.6
DATA,     483, 20090529 19:00:58,        5, STAT, gcs messages sent                       ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:58,        5, STAT, free buffer requested                   ,             1,          0,          1,         .2
DATA,     483, 20090529 19:00:58,        5, WAIT, gc cr block lost                        ,       5286330,    1057266,      5.29s,      1.06s
DATA,     483, 20090529 19:00:58,        5, WAIT, events in waitclass Other               ,           636,        127,      636us,    127.2us
--  End of snap 9
DATA,     483, 20090529 19:01:03,        5, STAT, cluster wait time                       ,           489,         98,        489,       97.8
DATA,     483, 20090529 19:01:03,        5, WAIT, gc cr request                           ,      10886698,    2177340,     10.89s,      2.18s
--  End of snap 10

PL/SQL procedure successfully completed.

.
Another graph of DBA_HIST_ACTIVE_SESS_HISTORY will show that the server is suffering from “gc cr block lost” and “gc cr multi block request”. Again, ADDM and AWR reports also support this graph

Hmm… I already have sufficient info that on the instance and session level, I’m getting waits that point me to diagnose and look into the network interconnect. Analyzing the “netstat -s” output from OS Watcher shows that there are many errors on the TCP and UDP packets

$ cat server1_netstat.dat | egrep -i "udpInOverflows|packet receive errors|fragments dropped|reassembles failed|fragments dropped after timeout"
    34096 fragments dropped after timeout
    306030 packet reassembles failed
    15 packet receive errors
    34096 fragments dropped after timeout
    306268 packet reassembles failed
    15 packet receive errors
    34096 fragments dropped after timeout
    306574 packet reassembles failed
    15 packet receive errors
    34096 fragments dropped after timeout
    306930 packet reassembles failed
    15 packet receive errors
    34096 fragments dropped after timeout

… output snipped … 

    15 packet receive errors
    34096 fragments dropped after timeout
    328087 packet reassembles failed
    15 packet receive errors
    34096 fragments dropped after timeout
    328493 packet reassembles failed
    15 packet receive errors
    34096 fragments dropped after timeout
    328889 packet reassembles failed
    15 packet receive errors

.
We quickly advised the network team to check on the interconnect switch for any issues and if possible replace it with a new unit to see if it solves the performance problem.

Then the DBA scheduled a downtime, and at first we tried restarting just the switch if it solves the problem. Then we make them run the applications again, but still there is a performance problem.

Then we shutdown again and the network team replaced the interconnect switch with a new unit. Then startup, and run the applications again…..

tentenenenen tenen!!!

…… they got fast :D !!!

Another check on the “netstat -s” shows no errors!

karao@karl:~/Desktop$ cat karlarao.dat | egrep -i "udpInOverflows|packet receive errors|fragments dropped|reassembles failed|fragments dropped after timeout"
    0 packet receive errors
    0 packet receive errors
    0 packet receive errors
    0 packet receive errors
    0 packet receive errors
    0 packet receive errors
    0 packet receive errors
    0 packet receive errors
    0 packet receive errors
    0 packet receive errors
    0 packet receive errors

.
Another graph of DBA_HIST_ACTIVE_SESS_HISTORY shows that the “gc block lost” waits are gone :D

3d view:

Stacked graph:

Update:
————–
I was also able to share and present this performance scenario on one of our DBA round tables locally (at the Oracle office)..
– Performance Scenario – Sudden slow down on RAC (pdf) – Oracle Philippines DBA round table, 07/21/2009

↧