Advanced Oracle Troubleshooting by Tanel Poder in Singapore

When I first saw that Tanel will conduct his seminar in Singapore, I told myself that I would even spend my own money just to be on that training! I’ve already read performance books like Optimizing Oracle Performance, Oracle 8i Internal Services, Forecasting Oracle Performance… And after that I still want more, and I still have questions that need to be answered. Well, if you’re on a tight budget you just opt to download some more docs/books to do multiple reads coupled with research/test cases and also reading through others blog…
But thanks to my boss for the funding, I was there!

The seminar has 8 chapters,

1) End to end lifecycle of a database request (33 slides)
2) Oracle, OS, and hardware interaction (39 slides)
3) Oracle kernel core services (39 slides)
4) OWI and kernel instrumentation framework (34 slides)
5) Lifecycle of a cursor (27 slides)
6) SQL execution row source operators and data flow (34 slides)
7) Logical and Physical IO internals (37 slides)
8 ) Lifecycle of a transaction (32 slides)

On the first day we were able to tackle the three chapters…

The 1st was about how to “Systematically Troubleshoot” performance problems. That Oracle is just a program and every execution that is happening is not magic, using the right tool for the right problem you’ll be able to pinpoint the root cause. Also he emphasized on how important it is to make the user’s session a starting point on diagnosis and he demo’d on how to make use of v$system_wait, snapper, ASH, v$sesstat, and of course if the built-in instrumentation is not enough… the process stack sampling is very useful that will tell you what kernel function where most of the time is spent…he also taught us how to read the stack tree using that magic Metalink note (175982.1)

2nd, dives into the OS troubleshooting, how you’ll systematically approach memory and CPU starvation on both Oracle and OS drilling down to the processes using Oracle (heapdump) and OS tools (ps, pstack, strace, procmon.exe for Windows – yeah you could also do process stack samples, pmap, top, vmstat)

3rd, was more of the internals where he discussed the core services of Oracle Kernel starting with the KGH module, the shared pool, and some more heaps, heap dumps and state objects, some X$ tables, and of course the most important part of the chapter… the Latching. He demo’d on how to systematically troubleshoot latching problems using latchprof, latchprofx, and snapper.

Then… At the end of the day, I was able to discuss with Tanel the concept of “performance visualization” see links below. I mean, yeah you can do reactive tuning and be very good at internals… but it doesn’t stop there… you have to be proactive, and visualizing/trending your performance numbers will help you on that. I’ve read his paper with Neil Gunther about how Multidimensional Visualization helps on analysing Oracle session level performance data that sometimes the usual x-y charts could mislead. He showed me some cool 3d performance monitoring tools running on JVM which is still on the works, but it’s really cool

And we also discussed about “capacity planning” because Neil Gunther is the author of the “Super Serial Scalability model” which he worked with (on the Visualization CMG paper) and Craig Shallahamer discussed it on the Chapter 10 (Scalability) of his book which is very interesting. Well, Tanel will still have to find out how efficient or realistic the “forecasting models” will be when applied with Oracle performance data. Well me too! But it still trickles down to response time and visualization

The second day…

We started with the Oracle Wait Interface, he answered my question on how would you know if it’s a syscall? Usercall? Or kernel function call?… he just used pstack, pmap, and the magic Metalink note. Cool Then he explained why the v$session is the most important view in 10g above. He showed us how to pinpoint if it’s a pl/sql that is executing or just plain SQL; yeah he has scripts for that! … the “sample” script is very useful for finding out which SQLs are problematic. He ended the chapter with identifying the session whether it is client-server or N-tiered application and demo’d on how to make use of dbms_session and dbms_monitor and how to trace these sessions.

5th discussed about lifecycle of cursors. Tanel discussed the internals of Library Cache and how applications live and die by how you do cursor management He also got good scripts here!

6th for me is the most interesting, this is where you start to glue together the info about the Oracle and OS interaction from day1 that Oracle is just a big complex C program… he started the chapter by clearly explaining how to properly get and read the explain plan (root -> branch -> leaves and row source -> filters -> access path) and associating each line with the Oracle kernel functions which you could get by doing a process stack and feeding the output to his “os_explain” tool…cool Then he ended the chapter with SQL plan profiling using some of his formatted explain plan scripts…

Unfortunately we ran out of time… we still have two chapters left! Tanel have to catch a flight to Sydney at 8pm. But he told us he will be blogging more about the later chapters, and at least we have his module… So the last demo was his “perfsheet” a very good visualization tool. Nice job with the pivot tables and 3d charts!

All in all, the popular geek from Estonia showed me how Oracle really works! (well, you owe us two chapters :p)

BTW, Porus Homi Havewala (http://enterprise-manager.blogspot.com) was also on the class…

Below are some of the seminar pictures: