LINK DOWNLOAD MIỄN PHÍ TÀI LIỆU "Tài liệu Optimizing Oracle Performance Section doc": http://123doc.vn/document/1043644-tai-lieu-optimizing-oracle-performance-section-doc.htm
Book: Optimizing Oracle Performance
Section: Chapter 1. A Better Way to Optimize
1.2 Requirements of a Good Method
What distinguishes a good method from a bad one? When we started hotsos.com in 1999, I began spending a lot of
time identifying the inefficiencies of existing Oracle performance improvement methods. It was a fun exercise. After
much study, my colleagues and I were able to construct a list of objectively measurable criteria that would assist in
distinguishing good from bad
in a method. We hoped that such a list would serve as a yardstick that would allow us to
measure the effectiveness of any method refinements we would create. Here is the list of attributes that I believe
distinguish good methods from bad ones:
Impact
If it is possible to improve performance, a method must deliver that improvement. It is unacceptable for a
performance remedy to require significant investment input but produce imperceptible or negative end-user
impact.
Efficiency
A method must always deliver performance improvement results with the least possible economic sacrifice. A
performance improvement method is not optimal if another method could have achieved a suitable result less
expensively in equal or less time.
Measurability
A method must produce performance improvement results that can be measured in units that make sense to the
business. Performance improvement measurements are inadequate if they can be expressed only in technical
units that do not correspond directly to improvement in cash flow, net profit, and return on investment.
Predictive capacity
A method must enable the analyst to predict the impact of a proposed remedy action. The unit of measure for
the prediction must be the same as that which the business will use to measure performance improvement.
Reliability
A method must identify the correct root cause of the problem, no matter what that root cause may be.
Determinism
A method must guide the analyst through an unambiguous sequence of steps that always rely upon
documented axioms, not experience or intuition. It is unacceptable for two analysts using the same method to
draw different conclusions about the root cause of a performance problem.
Finiteness
Page
1
of
2
O'Reilly Network Safari Bookshelf
-
Optimizing Oracle Performance
4/26/2004
http://safari.oreilly.com/?x=1&mode=print&sortKey=title&sortOrder=asc&view=&xmlid
www.it-ebooks.info
URL
http://safari.oreilly.com/059600527X/optoraclep-CHP-1-SECT-2
A method must have a well-defined terminating condition, such as a proof of optimality.
Practicality
A method must be usable in any reasonable operating condition. For example, it is unacceptable for a
performance improvement method to rely upon tools that exist in some operating environments but not others.
Method C suffers brutally on every single dimension of this eight-point definition of "goodness." I won't belabor the
point here, but I do encourage you to consider, right now, how your existing performance improvement methods score
on each of the attributes listed here. You might find the analysis quite motivating. When you've finished reading Part I
of this book, I hope you will revisit this list and see whether you think your scores have improved as a result of what
you have read.
Page
2
of
2
O'Reilly Network Safari Bookshelf
-
Optimizing Oracle Performance
4/26/2004
http://safari.oreilly.com/?x=1&mode=print&sortKey=title&sortOrder=asc&view=&xmlid
www.it-ebooks.info
Book: Optimizing Oracle Performance
Section: Chapter 1. A Better Way to Optimize
1.3 Three Important Advances
In the Preface, I began with the statement:
Optimizing Oracle response time is, for the most part, a solved problem.
This statement stands in stark contrast to the gloomy picture I painted at the beginning of this chapter—that, "For
many people, Oracle system performance is a very difficult problem." The contrast, of course, has a logical
explanation. It is this:
Several technological advances have added impact, efficiency, measurability, predictive capacity,
reliability, determinism, finiteness, and practicality to the science of Oracle performance optimization.
In particular, I believe that three important advances are primarily responsible for the improvements we have today.
Curiously, while these advances are new to most professionals who work with Oracle products, none of these
advances is really "new." Each is used extensively by optimization analysts in non-Oracle fields; some have been in
use for over a century.
1.3.1 User Action Focus
The first important advance in Oracle optimization technology follows from a simple mathematical observation:
You can't extrapolate detail from an aggregate.
Here's a puzzle to demonstrate my point. Imagine that I told you that a collection of 1,000 rocks contains 999 grey
rocks and one special rock that's been painted bright red. The collection weighs 1,000 pounds. Now, answer the
following question: "How much does the red rock weigh?" If your answer is, "I know that the red rock weighs one
pound," then, whether you realize it or not, you've told a lie. You don't know that the red rock weighs one pound.
With the information you've been given, you can't know. If your answer is, "I assume that the red rock weighs one
pound," then you're too generous in what you're willing to assume. Such an assumption puts you at risk of forming
conclusions that are incorrect—perhaps even stunningly incorrect.
The correct answer is that the red rock can weigh virtually any amount between zero and 1,000 pounds. The only
thing limiting the low end of the weight is the definition of how many atoms must be present in order for a thing to be
called a rock. Once we define how small a rock can be, then we've defined the high end of our answer. It is 1,000
pounds minus the weight of 999 of the smallest possible rocks. The red rock can weigh virtually anything between
zero and a thousand pounds. Answering with any more precision is wrong unless you happen to be very lucky. But
being very lucky at games like this is a skill that can be neither learned nor taught, nor repeated with acceptable
reliability.
This is one reason why Oracle analysts find it so frustrating to diagnose performance problems armed only with
system-wide statistics such as those produced by Statspack (or any of its cousins derived from the old SQL scripts
called bstat and estat). Two analysts looking at exactly the same Statspack output can "see" two completely different
things, neither of which is completely provable or completely disprovable by the Statspack output. It's not Statspack's
fault. It's a problem that is inherent in any performance analysis that uses system-wide data as its starting point
(V$SYSSTAT, V$SYSTEM_EVENT, and so on). You can in fact instruct Statspack to collect sufficiently granular data for
you, but no
Statspack documentation of which I'm aware makes any effort to tell you why you might ever want to.
A fine illustration is the case of an Oracle system whose red rock was a payroll processing problem. The officers of
the company described a performance problem with Oracle Payroll that was hurting their business. The database
administrators of the company described a performance problem with latches: cache buffers chains latches, to be
Page
1
of
4
O'Reilly Network Safari Bookshelf
-
Optimizing Oracle Performance
4/26/2004
http://safari.oreilly.com/?x=1&mode=print&sortKey=title&sortOrder=asc&view=&xmlid
www.it-ebooks.info
specific. Both arguments were compelling. The business truly was suffering from a problem with payroll being too
slow. You could see it, because checks weren't coming out of the system fast enough. The "system" truly was
suffering from latch contention problems. You could see it, because queries of V$SYSTEM_EVENT clearly showed that
the system was spending a lot of time waiting for the event called latch free.
The company's database and system administration staff had invested three frustrating months trying to fix the "latch
free problem," but the company had found no relief for the payroll performance problem. The reason was simple:
payroll wasn't spending time waiting for latches. How did we find out? We acquired operational timing data for one
execution of the slow payroll program. What we found was amazing. Yes, lots of other application programs in fact
spent time waiting to acquire cache buffers chains latches. But of the slow payroll program's total 1,985.40-second
execution time, only 23.69 seconds were consumed waiting on latches. That's 1.2% of the program's total response
time. Had the company completely eradicated waits for latch free from the face of their system, they would have made
only a 1.2% performance improvement in the response time of their payroll program.
How could system-wide statistics have been so misleading? Yes, lots of non-payroll workload was prominently
afflicted by latch free problems. But it was a grave error to assume that the payroll program's problem was the same as
the system-wide average problem. The error in assuming a cause-effect relationship between latch free waiting and
payroll performance cost the company three months of wasted time and frustration and thousands of dollars in labor
and equipment upgrade costs. By contrast, diagnosing the real payroll performance problem consumed only about ten
minutes of diagnosis time once the company saw the correct diagnostic data.
My colleagues and I encounter this type of problem repeatedly. The solution is for you (the performance analyst) to
focus entirely upon the user actions that need optimizing. The business can tell you what the most important user
actions are. The system cannot. Once you have identified a user action that requires optimization, then your first job is
to collect operational data exactly for that user action—no more, and no less.
1.3.2 Response Time Focus
For a couple of decades now, Oracle performance analysts have labored under the assumption that there's really no
objective way to measure Oracle response time [Ault and Brinson (2000), 27]. In the perceived absence of objective
ways to measure response time, analysts have settled for the next-best thing: event counts. And of course from event
counts come ratios. And from ratios come all sorts of arguments about which "tuning" actions are important, and
which ones are not.
However, users don't care about event counts and ratios and arguments; they care about response time: the duration
that begins when they request something and ends when they get their answer. No matter how much complexity you
build atop any timing-free event-count data, you are fundamentally doomed by the following inescapable truth, the
subject of the second important advance:
You can't tell how long something took by counting how many times it happened.
Users care only about response times. If you're measuring only event counts, then you're not measuring what the users
care about. If you liked the red rock quiz, here's another one for you: What's causing the performance problem in the
program that produced the data in Example 1-1?
Example 1-1. Components of response time listed in descending order of call volume
Response Time Component # Calls
CPU service 18,750
SQL*Net message to client 6,094
SQL*Net message from client 6,094
db file sequential read 1,740
log file sync 681
SQL*Net more data to client 108
SQL*Net more data from client 71
db file scattered read 34
direct path read 5
free buffer waits 4
log buffer space 2
Page
2
of
4
O'Reilly Network Safari Bookshelf
-
Optimizing Oracle Performance
4/26/2004
http://safari.oreilly.com/?x=1&mode=print&sortKey=title&sortOrder=asc&view=&xmlid
www.it-ebooks.info
direct path write 2
log file switch completion 1
latch free 1
Example 1-2 shows the same data from the same program execution, this time augmented with timing data (reported
in seconds) and sorted by descending response time impact. Does it change your answer?
Example 1-2. Components of response time listed in descending order of contribution to response time
Response Time Component Duration # Calls Dur/Call
SQL*Net message from client 166.6s 91.7% 6,094 0.027338s
CPU service 9.7s 5.3% 18,750 0.000515s
unaccounted-for 2.2s 1.2%
db file sequential read 1.6s 0.9% 1,740 0.000914s
log file sync 1.1s 0.6% 681 0.001645s
SQL*Net more data from client 0.3s 0.1% 71 0.003521s
SQL*Net more data to client 0.1s 0.1% 108 0.001019s
free buffer waits 0.1s 0.0% 4 0.022500s
SQL*Net message to client 0.0s 0.0% 6,094 0.000007s
db file scattered read 0.0s 0.0% 34 0.001176s
log file switch completion 0.0s 0.0% 1 0.030000s
log buffer space 0.0s 0.0% 2 0.005000s
latch free 0.0s 0.0% 1 0.010000s
direct path read 0.0s 0.0% 5 0.000000s
direct path write 0.0s 0.0% 2 0.000000s
Total 181.8s 100.0%
Of course it changes your answer, because response time is dominatingly important, and event counts are
inconsequential by comparison. The problem with the program that generated this data is what's going on with
SQL*Net message from client, not what's going on with CPU service.
If the year were 1991, we'd be in big trouble right now, because in 1991 the data that I've shown in this second table
wasn't available from the Oracle kernel. But if you've upgraded by now to at least Oracle7, then you don't need to
settle for event counts as the "next-
best thing" to response time data. The basic assumption that you can't tell how long
the Oracle kernel takes to do things is simply incorrect, and it has been since Oracle release 7.0.12.
1.3.3 Amdahl's Law
The final "great advance" in Oracle performance optimization that I'll mention is an observation published in 1967 by
Gene Amdahl, which has become known as Amdahl's Law [Amdahl (1967)]:
The performance enhancement possible with a given improvement is limited by the fraction of the
execution time that the improved feature is used.
In other words, performance improvement is proportional to how much a program uses the thing you improved.
Amdahl's Law is why you should view response time components in descending response time order. In Example 1-2
,
it's why you don't work on the CPU service "problem" before figuring out the SQL*Net message from client problem. If
you were to reduce total CPU consumption by 50%, you'd improve response time by only about 2%. But if you could
reduce the response time attributable to SQL*Net message from client by the same 50%, you'll reduce total response time
by 46%. In
Example 1-2, each percentage point of reduction in
SQL*Net message from client
duration produces nearly
twenty times the impact of a percentage point of CPU service reduction.
If you are an experienced Oracle performance analyst, you may have heard that SQL*Net
message from client
is an idle event that can be ignored. You must not ignore the so-called
idle events if you collect your diagnostic data in the manner I describe in Chapter 3.
Page
3
of
4
O'Reilly Network Safari Bookshelf
-
Optimizing Oracle Performance
4/26/2004
http://safari.oreilly.com/?x=1&mode=print&sortKey=title&sortOrder=asc&view=&xmlid
www.it-ebooks.info
URL
http://safari.oreilly.com/059600527X/optoraclep-CHP-1-SECT-3
Amdahl's Law is a formalization of optimization common sense. It tells you how to get the biggest "bang for the
buck" from your performance improvement efforts.
1.3.4 All Together Now
Combining the three advances in Oracle optimization technology into one statement results in the following simple
performance method:
Work first to reduce the biggest response time component of a business' most important user action.
It sounds easy, right? Yet I can be almost certain that this is not how you optimize your Oracle system back home. It's
not what your consultants do or what your tools do. This way of "tuning" is nothing like what your books or virtually
any of the other papers presented at Oracle seminars and conferences since 1980 tell you to do. So what is the missing
link?
The missing link is that unless you know how to extract and interpret response time measurements from your Oracle
system, you can't implement this simple optimization method. Explaining how to extract and interpret response time
measurements from your Oracle system is a main point of this book.
I hope that by the time you read this book, my claims that "this is not how you do it
today" don't make sense anymore. As I write this chapter, many factors are converging to
make the type of optimization I'm describing in this book much more common among
Oracle practitioners. If the book you're holding has played an influencing role in that
evolution, then so much the better.
Page
4
of
4
O'Reilly Network Safari Bookshelf
-
Optimizing Oracle Performance
4/26/2004
http://safari.oreilly.com/?x=1&mode=print&sortKey=title&sortOrder=asc&view=&xmlid
www.it-ebooks.info
Book: Optimizing Oracle Performance
Section: Chapter 1. A Better Way to Optimize
1.4 Tools for Analyzing Response Time
The definition of response time set forth by the International Organization for Standardization is plain but useful:
Response time is the elapsed time between the end of an inquiry or demand on a computer system and
the beginning of a response; for example, the length of the time between an indication of the end of an
inquiry and the display of the first character of the response at a user terminal (source:
http://searchnetworking.techtarget.com/sDefinition/0,,sid7_gci212896,00.html).
Response time is an objective measure of the interaction between a consumer and a provider. Consumers of computer
service want the right answer with the best response time for the lowest cost. Your goal as an Oracle performance
analyst is to minimize response time within the confines of the system owner's economic constraints. The ways to do
that become more evident when you consider the components of response time.
1.4.1 Sequence Diagram
A sequence diagram is a convenient way to depict the response time components of a user action. A sequence
diagram shows the flow of control as a user action consumes time in different layers of a technology stack. The
technology stack is a model that considers system components such as the business users, the network, the application
software, the database kernel, and the hardware in a stratified architecture. The component at each layer in the stack
demands service from the layer beneath it and supplies service to the layer above it. Figure 1-1 shows a sequence
diagram for a multi-tier Oracle system.
Figure 1-1. A sequence diagram for a multi-tier Oracle system
Figure 1-1 denotes the following sequence of actions, allowing us to literally see how each layer in the technology
stack contributes to the consumption of response time:
1.
After considering what she wants from the system, a user initiates a request for data from a browser by
pressing the OK button. Almost instantaneously, the request arrives at the browser. The user's perception of
response time begins with the click of the OK button.
Page
1
of
6
O'Reilly Network Safari Bookshelf
-
Optimizing Oracle Performance
4/26/2004
http://safari.oreilly.com/?x=1&mode=print&sortKey=title&sortOrder=asc&view=&xmlid
www.it-ebooks.info
2.
After devoting a short bit of time to rendering the pixels on the screen to make the OK button look like it has
been depressed, the browser sends an HTTP packet to the wide-area network (WAN). The request spends
some time on the WAN before arriving at the application server.
3.
After executing some application code on the middle tier, the application server issues a database call via
SQL*Net across the local-
area network (LAN). The request spends some time on the LAN (less than a request
across a WAN) before arriving at the database server.
4.
After consuming some CPU time on the database server, the Oracle kernel process issues an operating system
function call to perform a read from disk.
5.
After consuming some time in the disk subsystem, the read call returns control of the request back to the
database CPU.
6.
After consuming more CPU time on the database server, the Oracle kernel process issues another read request.
7.
After consuming some more time in the disk subsystem, the read call returns control of the request again to the
database CPU.
8.
After a final bit of CPU consumption on the database server, the Oracle kernel process passes the results of the
application server's database call. The return is issued via SQL*Net across the LAN.
9.
After the application server process converts the results of the database call into the appropriate HTML, it
passes the results to the browser across the WAN via HTTP.
10.
After rendering the result on the user's display device, the browser returns control of the request back to the
user. The user's perception of response time ends when she sees the information she requested.
In my opinion, the ideal Oracle performance optimization tool does not exist yet. The graphical user interface of the
ideal performance optimization tool would be a sequence diagram that could show how every microsecond of
response time had been consumed for any specified user action. Such an application would have so much information
to manage that it would have to make clever use of summary and drill-down features to show you exactly what you
wanted when you wanted it.
Such an application will probably be built soon. As you shall see throughout this book, much of the information that
is needed to build such an application is already available from the Oracle kernel. The biggest problems today are:
Most of the non-database tiers in a multi-tier system aren't instrumented to provide the type of response time
data that the Oracle kernel provides. Chapter 7 details the response time data that I'm talking about.
Depending upon your application architecture, it can be very difficult to collect properly scoped performance
diagnostic data for a specific user action. Chapter 3 explains what constitutes proper scoping for diagnostic
data, and Chapter 6 explains how to work around the data collection difficulties presented by various
application architectures.
However, much of what we need already exists. Beginning with Oracle release 7.0.12, and improving ever since, the
Oracle kernel is well instrumented for response time measurement. This book will help you understand exactly how to
A good sequence diagram reveals only the amount of detail that is appropriate for the
analysis at hand. For example, to simplify the content of Figure 1-1, I have made no effort
to show the tiny latencies that occur within the Browser, Apps Server, and DB CPU tiers
as their operating systems' schedulers transition processes among running and ready to
run states. In some performance improvement projects, understanding this level of detail
will be vital. I describe the performance impact of such state transitions in Chapter 7.
Page
2
of
6
O'Reilly Network Safari Bookshelf
-
Optimizing Oracle Performance
4/26/2004
http://safari.oreilly.com/?x=1&mode=print&sortKey=title&sortOrder=asc&view=&xmlid
www.it-ebooks.info
take advantage of those measurements to optimize your approach to the performance improvement of Oracle systems.
1.4.2 Resource Profile
A complete sequence diagram for anything but a very simple user action would show so much data that it would be
difficult to use all of it. Therefore, you need a way to summarize the details of response time in a useful way. In
Example 1-2, I showed a sample of such a summary, called a resource profile. A resource profile is simply a table
that reveals a useful decomposition of response time. Typically, a resource profile reveals at least the following
attributes:
Response time category
Total duration consumed by actions in that category
Number of calls to actions in that category
A resource profile is most useful when it lists its categories in descending order of elapsed time consumption per
category. The resource profile is an especially handy format for performance analysts because it focuses your
attention on exactly the problem you should solve first. The resource profile is the most important tool in my
performance diagnostic repertory.
The idea of the resource profile is nothing new, actually. The idea for using the resource profile as our company's
focus was inspired by an article on profilers published in the 1980s [Bentley (1988) 3-13], which itself was based on
work that Donald Knuth published in the early 1970s [Knuth (1971)]. The idea of decomposing response time into
components is so sensible that you probably do it often without realizing it. Consider how you optimize your driving
route to your favorite destination. Think of a "happy place" where you go when you want to feel better. For me it's my
local Woodcraft Supply store (http://www.woodcraft.com), which sells all sorts of tools that can cut fingers or crush
rib cages, and all sorts of books and magazines that explain how not to.
If you live in a busy city and schedule the activity during rush-hour traffic, the resource profile for such a trip might
resemble the following (expressed in minutes):
Response Time Component Duration # Calls Dur/Call
rush-hour expressway driving 90m 90% 2 45m
neighborhood driving 10m 10% 2 5m
Total 100m 100%
If the store were, say, only fifteen miles away, you might find the prospect of sitting for an hour and a half in rush-
hour traffic to be disappointing. Whether or not you believe that your brain works in the format of a resource profile,
you probably would consider the same optimization that I'm thinking of right now: perhaps you could go to the store
during an off-peak driving period.
Response Time Component Duration # Calls Dur/Call
off-peak expressway driving 30m 75% 2 15m
neighborhood driving 10m 25% 2 5m
Total 40m 100%
The driving example is simple enough, and the stakes are low enough, that a formal analysis is almost definitely
unnecessary. However, for more complex performance problems, the resource profile provides a convenient format
for proving a point, especially when decisions about whether or not to invest lots of time and money are involved.
Resource profiles add unequivocal relevance to Oracle performance improvement projects.
Example 1-3 shows a
resource profile for the Oracle Payroll program described earlier in Section 1.3.1. Before the database administrators
saw this resource profile, they had worked for three months fighting a perceived problem with latch contention. In
desperation, they had spent several thousand dollars on a CPU upgrade, which had actually degraded the response
Page
3
of
6
O'Reilly Network Safari Bookshelf
-
Optimizing Oracle Performance
4/26/2004
http://safari.oreilly.com/?x=1&mode=print&sortKey=title&sortOrder=asc&view=&xmlid
www.it-ebooks.info
time of the payroll action whose performance they were trying to improve. Within ten minutes of creating this
resource profile, the database administrator knew exactly how to cut this program's response time by roughly 50%.
The problem and its solution are detailed in Part II of this book.
Example 1-
3. The resource profile for a network configuration problem that had previously been misdiagnosed
as both a latch contention problem and a CPU capacity problem
Response Time Component Duration # Calls Dur/Call
SQL*Net message from client 984.0s 49.6% 95,161 0.010340s
SQL*Net more data from client 418.8s 21.1% 3,345 0.125208s
db file sequential read 279.3s 14.1% 45,084 0.006196s
CPU service 248.7s 12.5% 222,760 0.001116s
unaccounted-for 27.9s 1.4%
latch free 23.7s 1.2% 34,695 0.000683s
log file sync 1.1s 0.1% 506 0.002154s
SQL*Net more data to client 0.8s 0.0% 15,982 0.000052s
log file switch completion 0.3s 0.0% 3 0.093333s
enqueue 0.3s 0.0% 106 0.002358s
SQL*Net message to client 0.2s 0.0% 95,161 0.000003s
buffer busy waits 0.2s 0.0% 67 0.003284s
db file scattered read 0.0s 0.0% 2 0.005000s
SQL*Net break/reset to client 0.0s 0.0% 2 0.000000s
Total 1,985.4s 100.0%
Example 1-4
shows another resource profile that saved a project from a frustrating and expensive ride down a rat hole.
Before seeing the resource profile shown here, the proposed solution to this report's performance problem was to
upgrade either memory or the I/O subsystem. The resource profile proved unequivocally that upgrading either could
result in no more than a 2% response time improvement. Almost all of this program's response time was attributable
to a single SQL statement that motivated nearly a billion visits to blocks stored in the database buffer cache.
Problems like this are commonly caused by operational errors like the accidental deletion of schema statistics used by
the Oracle cost-based query optimizer (CBO).
Example 1-4. The resource profile for an inefficient SQL problem that had previously been diagnosed as an
I/O subsystem problem
Response Time Component Duration # Calls Dur/Call
CPU service 48,946.7s 98.0% 192,072 0.254835s
db file sequential read 940.1s 2.0% 507,385 0.001853s
SQL*Net message from client 60.9s 0.0% 191,609 0.000318s
latch free 2.2s 0.0% 171 0.012690s
other 1.4s 0.0%
Total 49,951.3s 100.0%
Example 1-4 is a beautiful example of how a resource profile can free you from victimization to myth. In this case,
the myth that had confused the analyst about this slow session was the proposition that a high database buffer cache
hit ratio is an indication of SQL statement efficiency. The statement causing this slow session had an exceptionally
high buffer cache hit ratio. It is easy to understand why, by looking at the computation of the cache hit ratio (CHR)
metric for this case:
You can't tell by looking at the resource profile in Example 1-4 that the CPU capacity was
consumed by nearly a billion memory reads. Each of the 192,072 "calls" to the CPU service
resource represents one Oracle database call (for example, a parse, an execute, or a fetch).
From the detailed SQL trace information collected for each of these calls, I could
determine that the 192,072 database calls had issued nearly a billion memory reads. How
you can do this is detailed in Chapter 5.
Page
4
of
6
O'Reilly Network Safari Bookshelf
-
Optimizing Oracle Performance
4/26/2004
http://safari.oreilly.com/?x=1&mode=print&sortKey=title&sortOrder=asc&view=&xmlid
www.it-ebooks.info
Không có nhận xét nào:
Đăng nhận xét