123
 123

Tip: 看不到本站引用 Flickr 的图片? 下载 Firefox Access Flickr 插件 | AD: 订阅 DBA notes --

2018-10-18 Thu

06:33 Welcome (2717 Bytes) » Guy Harrison's main page

This site contains links to material relevent to my professional life. I'm an IT professional with experience in a range of disciplines and technologies but probably best known for my involvement in databases such as Oracle, MongoDB and MySql and for my writings on emerging database technologies.

I'm CTO of Southbank Software where we create the open source database development tool dbKoda.  I'm also a partner at Toba Capital 

I'm the author of Next Generation Databases,  Oracle Performance Survival Guide, MySql Stored Procedure Programming (with Steven Feuerstein), Oracle SQL High Performance Tuning and Oracle Desk Reference, as well as contributing to several other books on database technology. I write a monthly columns at Database Trends and Applications and more occasional articles elsewhere. 

I'm an Oracle ACE alumni and a MongoDB certified DBA and developer. 

My blog contains occasional contributions of a technical nature and I can be contacted by email either at guy.a.harrison@gmail.com.  I'm @guyharrison on twitter.

I live in Melbourne Australia with my wife Jenny, daughter Emily, one dog and one cat in beautiful Yarraville.    My adult children Chris, Kate and Michael and their partners, dogs, cats and killer rabbits live nearby. 

In my spare time I worry about what on earth I'd do if I ever had any spare time. 

06:33 “How did you learn so much stuff about Oracle?” (9356 Bytes) » Cary Millsap
In LinkedIn, a new connection asked me a very nice question. He asked, “I know this might sound stupid, but how did you learn so much stuff about Oracle. :)”

Good one. I like the presumption that I know a lot of stuff about Oracle. I suppose that I do, at least about some some aspects of it, although I often feel like I don’t know enough. It occurred to me that answering publicly might also be helpful to anyone trying to figure out how to prepare for a career. Here’s my answer.

I took a job with the young consulting division of Oracle Corporation in September 1989, about two weeks after the very first time I had heard the word “Oracle” used as the name of a company. My background had been mathematics and computer science in school. I had two post-graduate degrees: a Master of Science Computer Science with a focus on language design and compilers, and a Master of Business Administration with a focus in finance.

My first “career job” was as a software engineer, which I started before the MBA. I designed languages and wrote compilers to implement those languages. Yes, people actually pay good money for that, and it’s possibly still the most fun I’ve ever had at work. I wrote software in C, lex, and yacc, and I taught my colleagues how to do it, too. In particular, I spent a lot of time teaching my colleagues how to make their C code faster and more portable (so it would run on more computers than just the one on which you wrote it).

Even though I loved my job, I didn’t see a lot of future in it. At least not in Colorado Springs in the late 1980s. So I took a year off to get the MBA at SMU in Dallas. I went for the MBA because I thought I needed to learn more about money and business. It was the most difficult academic year of my life, because I was not particularly connected to or even interested in most of the subject matter. I hated a lot of my classes, which made it difficult to do as well as I had been accustomed. But I kept grinding away, and finished my degree in the year it was supposed to take. Of course I learned many, many things that year that have been vital to my career.

A couple of weeks after I got my MBA, I went to work for Oracle in Dallas, with a salary that was 168% of what it had been as a compiler designer. My job was to visit Oracle customers and help them with their problems.

It took a while for me to get into a good rhythm at Oracle. My boss was sending me to these local customers that were having problems with the Oracle Financial Applications (the “Finapps,” as we usually called them, which would many years later become the E-Business Suite) on version 6.0.26 of the ORACLE database (it was all caps back then). At first, I couldn’t help them near as much as I had wanted to. It was frustrating.

That actually became my rhythm: week after week, I visited these people who were having horrific problems with ORACLE and the Finapps. The database in 1990, although it had some pretty big bugs, was still pretty good. It was the applications that caused most of the problems I saw. There were a lot of problems, both with the software and with how it was sold. My job was to fix the problems. Some of those problems were technical. Many were not.

A lot of the problems were performance; problems of the software running “too slowly.” I found those problems particularly interesting. For those, I had some experience and tools at my disposal. I knew a good bit about operating systems and compilers and profilers and linkers and debuggers and all that, and so learning about Oracle indexes and rollback segments (two good examples, continual sources of customer frustration) wasn’t that scary of a step for me.

I hadn’t learned anything about Oracle or relational databases in school, I learned about how the database worked at Oracle by reading the documentation, beginning with the excellent Oracle® Database Concepts. Oracle sped me along a bit with a couple of the standard DBA courses.

My real learning came from being in the field. The problems my customers had were immediately interesting by virtue of being important. The resources available to me for solving such problems back in the early 1990s were really just books, email, and the telephone. The Internet didn’t exist yet. (Can you imagine?) The Oracle books available back then, for the most part, were absolutely horrible. Just garbage. Just about the only thing they were good for was creating problems that you could bill lots of consulting hours to fix. The only thing that was left was email and the telephone.

The problem with email and telephones, however, is that there has to be someone on the other end. Fortunately, I had that. The people on the other end of my email and phone calls were my saviors and heroes. In my early Oracle years, those saviors and heroes included people like Darryl Presley, Laurel Jamtgaard, Tom Kemp, Charlene Feldkamp, David Ensor, Willis Ranney, Lyn Pratt, Lawrence To, Roderick Mañalac, Greg Doherty, Juan Loaiza, Bill Bridge, Brom Mahbod, Alex Ho, Jonathan Klein, Graham Wood, Mark Farnham (who didn’t even work for Oracle, but who could cheerfully introduce me to anyone I needed), Anjo Kolk, and Mogens Nørgaard. I could never repay these people, and many more, for what they did for me. ...In some cases, at all hours of the night.

So, how did I learn so much stuff about Oracle? It started by immersing myself into a universe where every working day I had to solve somebody’s real Oracle problems. Uncomfortable, but effective. I survived because I was persistent and because I had a great company behind me, filled with spectacularly intelligent people who loved helping each other. Could I have done that on my own, today, with the advent of the Internet and lots and lots of great and reliable books out there to draw upon? I doubt it. I sincerely do. But maybe if I were young again...

I tell my children, there’s only one place where money comes from: other people. Money comes only from other people. So many things in life are that way.

I’m a natural introvert. I naturally withdraw from group interactions whenever I don’t feel like I’m helping other people. Thankfully, my work and my family draw me out into the world. If you put me into a situation where I need to solve a technical problem that I can’t solve by myself, then I’ll seek help from the wonderful friends I’ve made.

I can never pay it back, but I can try to pay it forward.

(Oddly, as I’m writing this, I realize that I don’t take the same healthy approach to solving business problems. Perhaps it’s because I naturally assume that my friends would have fun helping solve a technical problem, but that solving a business problem would not be fun and therefore I would be imposing upon them if I were to ask for help solving one. I need to work on that.)

So, to my new LinkedIn friend, here’s my advice. Here’s what worked for me:
  • Educate yourself. Read, study, experiment. Educate yourself especially well in the fundamentals. So many people don’t. Being fantastic at the fundamentals is a competitive advantage, no matter what you do. If it’s Oracle you’re interested in learning about, that’s software, so learn about software: about operating systems, and C, and linkers, and profilers, and debuggers, .... Read the Oracle Database Concepts guide and all the other free Oracle documentation. Read every book there is by Tom Kyte and Christian Antognini and Jonathan Lewis and Tanel Põder and Kerry Osborne and Karen Morton and James Morle all the other great authors out there today. And read their blogs.
  • Find a way to hook yourself into a network of people that are willing and able to help you. You can do that online these days. You can earn your way into a community by doing things like asking thoughtful questions, treating people respectfully (even the ones who don’t treat you respectfully), and finding ways to teach others what you’ve learned. Write. Write what you know, for other people to use and improve. And for God’s sake, if you don’t know something, don’t act like you do. That just makes everyone think you’re an asshole, which isn’t helpful.
  • Immerse yourself into some real problems. Read Scuttle Your Ships Before Advancing if you don’t understand why. You can solve real problems online these days, too (e.g., StackExchange and even Oracle.com), although I think that it’s better to work on real live problems at real live customer sites. Stick with it. Fix things. Help people.
Help people.

That’s my advice.
06:33 Words I Don’t Use, Part 5: “Wait” (10127 Bytes) » Cary Millsap
The fifth “word I do not use” is the Oracle technical term wait.

The Oracle Wait Interface

In 1991, Oracle Corporation released some of the most important software instrumentation of all time: the “wait” statistics that were implemented in Oracle 7.0. Here’s part of the story, in Juan Loaiza’s words, as told in Nørgaard et. al (2004), Oracle Insights: Tales of the Oak Table.
This stuff was developed because we were running a benchmark that we could not get to perform. We had spent several weeks trying to figure out what was happening with no success. The symptoms were clear—the system was mostly idle—we just couldn’t figure out why.

We looked at the statistics and ratios and kept coming up with theories, the trouble was that none of them were right. So we wasted weeks tuning and fixing things that were not the problem. Finally we ran out of ideas and were forced to go back and instrument the code to figure out what the problem was.

Once the waits were instrumented the problem was diagnosed in minutes. We were having “free buffer” waits because the DBWR was not writing blocks fast enough. It’s amazing how hard that was to figure out with statistics, and how easy it was to figure out once the waits were instrumented.

...In retrospect a lot of the names could be greatly improved. The wait interface was added after the freeze date as a “stealth” project so it did not get as well thought through as it should have. Like I said, we were just trying to solve a problem in the course of a benchmark. The trouble is that so many people use this stuff now that if you change the names it will break all sorts of thing tools, so we have to leave them alone.
Before Juan’s team added this code, the Oracle kernel would show you only how much time its user calls (like parse, exec, and fetch) were taking. The new instrumentation, which included a set of new fixed views like v$session_wait and new WAIT lines in our trace files, showed how much time Oracle’s system calls (like reads, writes, and semops) were taking.

The Working-Waiting Model

The wait interface begat a whole new mental model about Oracle performance, based on the principle of working versus waiting:
Response Time = Service Time + Wait Time
In this formula, Oracle defines service time as the duration of the CPU used by your Oracle session (the duration Oracle spent working), and wait time as the sum of the durations of your Oracle wait events (the duration that Oracle spent waiting). Of course, response time in this formula means the duration spent inside the Oracle Database kernel.

Why I Don’t Say Wait, Part 1

There are two reasons I don’t use the word wait. The first is simply that it’s ambiguous.

The Oracle formula is okay for talking about database time, but the scope of my attention is almost never just Oracle’s response time—I’m interested in the business’s response time. And when you think about the whole stack (which, of course you do; see holistic), there are events we could call wait events all the way up and down:
  • The customer waits for an answer from a user.
  • The user waits for a screen from the browser.
  • The browser waits for an HTML page from the application server.
  • The application server waits for a database call from the Oracle kernel.
  • The Oracle kernel waits for a system call from the operating system.
  • The operating system’s I/O request waits to clear the device’s queue before receiving service.
  • ...
If I say waits, the users in the room will think I’m talking about application response time, the Oracle people will think I’m talking about Oracle system calls, and the hardware people will think I’m talking about device queueing delays. Even when I’m not.

Why I Don’t Say Wait, Part 2

There is a deeper problem with wait than just ambiguity, though. The word wait invites a mental model that actually obscures your thinking about performance.

Here’s the problem: waiting sounds like something you’d want to avoid, and working sounds like something you’d want more of. Your program is waiting?! Unacceptable. You want it to be working. The connotations of the words working and waiting are unavoidable. It sounds like, if a program is waiting a lot, then you need to fix it; but if it’s working a lot, then it is probably okay. Right?

Actually, no.

The connotations “work is virtuous” and “waits are abhorrent” are false connotations in Oracle. One is not inherently better or worse than the other. Working and waiting are not accurate value judgments about Oracle software. On the contrary, they’re not even meaningful; they’re just arbitrary labels. We could just as well have been taught to say that an Oracle program is “working on disk I/O” and “waiting to finish its CPU instructions.”

The terms working and waiting really just refer to different subroutine call types:

“Oracle is workingmeans“your Oracle kernel process is executing a user call”
“Oracle is waitingmeans“your Oracle kernel process is executing a system call”

The working-waiting model implies a distinction that does not exist, because these two call types have equal footing. One is no worse than the other, except by virtue of how much time it consumes. It doesn’t matter whether a program is working or waiting; it only matters how long it takes.

Working-Waiting Is a Flawed Analogy

The working-waiting paradigm is a flawed analogy. I’ll illustrate. Imagine two programs that consume 100 seconds apiece when you run them:

Program AProgram B
DurationCall typeDurationCall type
98system calls (waiting)98user calls (working)
2user calls (working)2system calls (waiting)
100Total100Total

To improve program A, you should seek to eliminate unnecessary system calls, because that’s where most of A’s time has gone. To improve B, you should seek to eliminate unnecessary user calls, because that’s where most of B’s time has gone. That’s it. Your diagnostic priority shouldn’t be based on your calls’ names; it should be based solely on your calls’ contributions to total duration. Specifically, conclusions like, “Program B is okay because it doesn’t spend much time waiting,” are false.

A Better Model

I find that discarding the working-waiting model helps people optimize better. Here’s how you can do it. First, understand the substitute phrasing: working means executing a user call; and waiting means executing a system call. Second, understand that the excellent ideas people use to optimize other software are excellent ideas for optimizing Oracle, too:
  1. Any program’s duration is a function of all of its subroutine call durations (both user calls and system calls), and
  2. A program is running as fast as possible only when (1) its unnecessary calls have been eliminated, and (2) its necessary calls are running at hardware speed.
Oracle’s wait interface is vital because it helps us measure an Oracle program’s complete execution duration—not just Oracle’s user calls, but its system calls as well. But I avoid saying wait to help people steer clear of the incorrect bias introduced by the working-waiting analogy.
06:33 Words I Don’t Use, Part 4: “Expert” (2240 Bytes) » Cary Millsap
The fourth “word I do not use” is expert.

When I was a young boy, my dad would sometimes drive me to school. It was 17 miles of country roads and two-lane highways, so it gave us time to talk.

At least once a year, and always on the first day of school, he would tell me, “Son, there are two answers to every test question. There’s the correct answer, and there’s the answer that the teacher expects. ...They’re not always the same.”

He would continue, “And I expect you to know them both.”

He wanted me to make perfect grades, but he expected me to understand my responsibility to know the difference between authority and truth. My dad thus taught me from a young age to be skeptical of experts.

The word expert always warns me of a potentially dangerous type of thinking. The word is used to confer authority upon the person it describes. But it’s ideas that are right or wrong; not people. You should evaluate an idea on its own merit, not on the merits of the person who conveys it. For every expert, there is an equal and opposite expert; but for every fact, there is not necessarily an equal and opposite fact.

A big problem with expert is corruption—when self-congratulators hijack the label to confer authority upon themselves. But of course, misusing the word erodes the word. After too much abuse within a community, expert makes sense only with finger quotes. It becomes a word that critical thinkers use only ironically, to describe people they want to avoid.
06:33 Words I Don’t Use, Part 3: “Best Practice” (1515 Bytes) » Cary Millsap
The third “word I do not use” is best practice.

The “best practice” serves a vital need in any industry. It is the answer to, “Please don’t make me learn about this; just tell me what to do.” The “best practice” is a fine idea in spirit, but here’s the thing: many practices labeled “best” don’t deserve the adjective. They’re often containers for bad advice.

The most common problem with “best practices” is that they’re not parameterized like they should be. A good practice usually depends on something: if this is true, then do that; otherwise, do this other thing. But most “best practices” don’t come with conditions of execution—they often contain no if statements at all. They come disguised as recipes that can save you time, but they often encourage you to skip past thinking about things that you really ought to be thinking about.

Most of my objections to “best practices” go away when the practices being prescribed are actually good. But the ones I see are often not, like the old SQL “avoid full-table scans” advice. Enforcing practices like this yields applications that don’t run as well as they should and developers that don’t learn the things they should. Practices like “Measure the efficiency of your SQL at every phase of the software life cycle,” are actually “best”-worthy, but alas, they’re less popular because they sound like real work.
06:33 Words I Don’t Use, Part 2: “Holistic” (728 Bytes) » Cary Millsap
The second “word I do not use” is holistic.

When people use the word “holistic” in my industry (Oracle), it means that they’re paying attention to not just an individual subcomponent of a system, but to a whole system, including (I hope) even the people it serves.

But trying to differentiate technology services by saying “we take a holistic view of your system” is about like differentiating myself by saying I’ll wear clothes to work. Saying “holistic” would make it look like I’ve only just recently become aware that optimizing a system’s individual subsystems is not a reliable way to optimize the system itself. This should not be a distinctive revelation.
06:33 Words I Don’t Use, Part 1: “Methodology” (1775 Bytes) » Cary Millsap
Today, I will begin a brief sequence of blog posts that I’ll call “Words I Don’t Use.” I hope you’ll have some fun with it, and maybe find a thought or two worth considering.

The first word I’ll discuss is methodology. Yes. I made a shirt about it.

Approximately 100% of the time in the [mostly non-scientific] Oracle world that I live in, when people say “methodology,” they’re using it in the form that American Heritage describes as a pretentious substitute for “method.” But methodology is not the same as method. Methods are processes. Sequences of steps. Methodology is the scientific study of methods.

I like this article called “Method versus Methodology” by Peter Klein, which cites the same American Heritage Dictionary passage that I quoted on page 358 of Optimizing Oracle Performance.
06:33 When is Video Better? (3242 Bytes) » Cary Millsap
Ok, I’m stuck, and I need your help.

At my company, we sell software tools that help Oracle application developers and database administrators see exactly where their code spends their users’ time. I want to publish better information at our web page that will allow people who are interested to learn more about our software, and that will allow people who don’t even realize we exist to discover what we have. My theory is that the more people who understand exactly what we have, the more customers we’ll get, and we have some evidence that bears that out.

I’ve gotten so much help from YouTube in various of my endeavors that I’ve formed the grand idea in my head:
We need more videos showing our products.
The first one I made is the 1:13 video on YouTube called “Method R Tools v3.0: Getting Started.” I’m interested to see how effective it is. I think this content is perfect for the video format because the whole point is to show you how easy it is to get going. Saying it’s easy just isn’t near as fun or convincing as showing it’s easy.

The next thing I need to share with people is a great demonstration that Jeff Holt has helped me pull together, which shows off all the tools in our suite, how they interact and solve an interesting and important common problem. But this one can’t be a 1-minute video; it’s a story that will take a lot longer to tell than that. It’s probably a 10- to 15-minute story, if I had to guess.

Here’s where I’m stuck. Should I make a video? Or write up a blog post for it? The reason this is a difficult question is that making a video costs me about 4 hours per minute of output I can create. That will get better over time, as I practice and accumulate experience. But right now, videos cost me a lot of time. On the other hand, I can whip together a blog post with plenty of detail in a fraction of the time.

Where I need your help is to figure out how much benefit there is, really, to creating a video instead of just a write-up. Some things I have to consider:
  • Would a 15-minute video capture the attention of people who Do people glaze over (TL;DR) on longish printed case studies on which they’d gladly watch about 15 minutes of video?
  • Or do people just pass on the prospect of watching a 15-minute case study about a problem they might not even realize they have yet? I ask this, because I find myself not clicking on any video that I know will take longer than a minute or two, unless I believe it’s going to help me solve a difficult problem that I know I have right now.
So, if you don’t mind giving me a hand, I’ve created a survey at SurveyMonkey that asks just three simple questions that will help me determine which direction to go next. I’m eager to see what you say.

Thank you for your time.
06:33 What happened to “when the application is fast enough to meet users’ requirements?” (17247 Bytes) » Cary Millsap
On January 5, I received an email called “Video” from my friend and former employee Guđmundur Jósepsson from Iceland. His friends call him Gummi (rhymes with “who-me”). Gummi is the guy whose name is set in the ridiculous monospace font on page xxiv of Optimizing Oracle Performance, apparently because O’Reilly’s Linotype Birka font didn’t have the letter eth (đ) in it. Gummi once modestly teased me that this is what he is best known for. But I digress...

His email looked like this:


It’s a screen shot of frame 3:12 from my November 2014 video called “Why you need a profiler for Oracle.” At frame 3:12, I am answering the question of how you can know when you’re finished optimizing a given application function. Gummi’s question is, «Oi! What happened to “when the application is fast enough to meet users’ requirements?”»

Gummi noticed (the good ones will do that) that the video says something different than the thing he had heard me say for years. It’s a fair question. Why, in the video, have I said this new thing? It was not an accident.

When are you finished optimizing?

The question in focus is, “When are you finished optimizing?” Since 2003, I have actually used three different answers:
When are you are finished optimizing?
  1. When the cost of call reduction and latency reduction exceeds the cost of the performance you’re getting today.
    Source: Optimizing Oracle Performance (2003) pages 302–304.
  2. When the application is fast enough to meet your users’ requirements.
    Source: I have taught this in various courses, conferences, and consulting calls since 1999 or so.
  3. When there are no unnecessary calls, and the calls that remain run at hardware speed.
    Source: “Why you need a profiler for Oracle” (2014) frames 2:51–3:20.
My motive behind answers A and B was the idea that optimizing beyond what your business needs can be wasteful. I created these answers to deter people from misdirecting time and money toward perfecting something when those resources might be better invested improving something else. This idea was important, and it still is.

So, then, where did C come from? I’ll begin with a picture. The following figure allows you to plot the response time for a single application function, whatever “given function” you’re looking at. You could draw a similar figure for every application function on your system (although I wouldn’t suggest it).


Somewhere on this response time axis for your given function is the function’s actual response time. I haven’t marked that response time’s location specifically, but I know it’s in the blue zone, because at the bottom of the blue zone is the special response time RT. This value RT is the function’s top speed on the hardware you own today. Your function can’t go faster than this without upgrading something.

It so happens that this top speed is the speed at which your function will run if and only if (i) it contains no unnecessary calls and (ii) the calls that remain run at hardware speed. ...Which, of course, is the idea behind this new answer C.

Where, exactly, is your “requirement”?

Answer B (“When the application is fast enough to meet your users’ requirements”) requires that you know the users’ response time requirement for your function, so, next, let’s locate that value on our response time axis.

This is where the trouble begins. Most DBAs don’t know what their users’ response time requirements really are. Don’t despair, though; most users don’t either.

At banks, airlines, hospitals, telcos, and nuclear plants, you need strict service level agreements, so those businesses investment into quantifying them. But realize: quantifying all your functions’ response time requirements isn’t about a bunch of users sitting in a room arguing over which subjective speed limits sound the best. It’s about knowing your technological speed limits and understanding how close to those values your business needs to pay to be. It’s an expensive process. At some companies, it’s worth the effort; at most companies, it’s just not.

How about using, “well, nobody complains about it,” as all the evidence you need that a given function is meeting your users’ requirement? It’s how a lot of people do it. You might get away with doing it this way if your systems weren’t growing. But systems do grow. More data, more users, more application functions: these are all forms of growth, and you can probably measure every one of them happening where you’re sitting right now. All these forms of growth put you on a collision course with failing to meet your users’ response time requirements, whether you and your users know exactly what they are, or not.

In any event, if you don’t know exactly what your users’ response time requirements are, then you won’t be able to use “meets your users’ requirement” as your finish line that tells you when to stop optimizing. This very practical problem is the demise of answer B for most people.

Knowing your top speed

Even if you do know exactly what your users’ requirements are, it’s not enough. You need to know something more.

Imagine for a minute that you do know your users’ response time requirement for a given function, and let’s say that it’s this: “95% of executions of this function must complete within 5 seconds.” Now imagine that this morning when you started looking at the function, it would typically run for 10 seconds in your Oracle SQL Developer worksheet, but now after spending an hour or so with it, you have it down to where it runs pretty much every time in just 4 seconds. So, you’ve eliminated 60% of the function’s response time. That’s a pretty good day’s work, right? The question is, are you done? Or do you keep going?

Here is the reason that answer C is so important. You cannot responsibly answer whether you’re done without knowing that function’s top speed. Even if you know how fast people want it to run, you can’t know whether you’re finished without knowing how fast it can run.

Why? Imagine that 85% of those 4 seconds are consumed by Oracle enqueue, or latch, or log file sync calls, or by hundreds of parse calls, or 3,214 network round-trips to return 3,214 rows. If any of these things is the case, then no, you’re absolutely not done yet. If you were to allow some ridiculous code path like that to survive on a production system, you’d be diminishing the whole system’s effectiveness for everybody (even people who are running functions other than the one you’re fixing).

Now, sure, if there’s something else on the system that has a higher priority than finishing the fix on this function, then you should jump to it. But you should at least leave this function on your to-do list. Your analysis of the higher priority function might even reveal that this function’s inefficiencies are causing the higher-priority functions problems. Such can be the nature of inefficient code under conditions of high load.

On the other hand, if your function is running in 4 seconds and (i) its profile shows no unnecessary calls, and (ii) the calls that remain are running at hardware speeds, then you’ve reached a milestone:
  1. if your code meets your users’ requirement, then you’re done;
  2. otherwise, either you’ll have to reimagine how to implement the function, or you’ll have to upgrade your hardware (or both).
There’s that “users’ requirement” thing again. You see why it has to be there, right?

Well, here’s what most people do. They get their functions’ response times reasonably close to their top speeds (which, with good people, isn’t usually as expensive as it sounds), and then they worry about requirements only if those requirements are so important that it’s worth a project to quantify them. A requirement is usually considered really important if it’s close to your top speed or if it’s really expensive when you violate a service level requirement.

This strategy works reasonably well.

It is interesting to note here that knowing a function’s top speed is actually more important than knowing your users’ requirements for that function. A lot of companies can work just fine not knowing their users’ requirements, but without knowing your top speeds, you really are in the dark. A second observation that I find particularly amusing is this: not only is your top speed more important to know, your top speed is actually easier to compute than your users’ requirement (…if you have a profiler, which was my point in the video).

Better and easier is a good combination.

Tomorrow is important, too

When are you are finished optimizing?
  1. When the cost of call reduction and latency reduction exceeds the cost of the performance you’re getting today.
  2. When the application is fast enough to meet your users’ requirements.
  3. When there are no unnecessary calls, and the calls that remain run at hardware speed.
Answer A is still a pretty strong answer. Notice that it actually maps closely to answer C. Answer C’s prescription for “no unnecessary calls” yields answer A’s goal of call reduction, and answer C’s prescription for “calls that remain run at hardware speed” yields answer A’s goal of latency reduction. So, in a way, C is a more action-oriented version of A, but A goes further to combat the perfectionism trap with its emphasis on the cost of action versus the cost of inaction.

One thing I’ve grown to dislike about answer A, though, is its emphasis on today in “…exceeds the cost of the performance you’re getting today.” After years of experience with the question of when optimization is complete, I think that answer A under-emphasizes the importance of tomorrow. Unplanned tomorrows can quickly become ugly todays, and as important as tomorrow is to businesses and the people who run them, it’s even more important to another community: database application developers.

Subjective goals are treacherous for developers

Many developers have no way to test, today, the true production response time behavior of their code, which they won’t learn until tomorrow. ...And perhaps only until some remote, distant tomorrow.

Imagine you’re a developer using 100-row tables on your desktop to test code that will access 100,000,000,000-row tables on your production server. Or maybe you’re testing your code’s performance only in isolation from other workload. Both of these are problems; they’re procedural mistakes, but they are everyday real-life for many developers. When this is how you develop, telling you that “your users’ response time requirement is n seconds” accidentally implies that you are finished optimizing when your query finishes in less than n seconds on your no-load system of 100-row test tables.

If you are a developer writing high-risk code—and any code that will touch huge database segments in production is high-risk code—then of course you must aim for the “no unnecessary calls” part of the top speed target. And you must aim for the “and the calls that remain run at hardware speed” part, too, but you won’t be able to measure your progress against that goal until you have access to full data volumes and full user workloads.

Notice that to do both of these things, you must have access to full data volumes and full user workloads in your development environment. To build high-performance applications, you must do full data volume testing and full user workload testing in each of your functional development iterations.

This is where agile development methods yield a huge advantage: agile methods provide a project structure that encourages full performance testing for each new product function as it is developed. Contrast this with the terrible project planning approach of putting all your performance testing at the end of your project, when it’s too late to actually fix anything (if there’s even enough budget left over by then to do any testing at all). If you want a high-performance application with great performance diagnostics, then performance instrumentation should be an important part of your feedback for each development iteration of each new function you create.

My answer

So, when are you finished optimizing?
  1. When the cost of call reduction and latency reduction exceeds the cost of the performance you’re getting today.
  2. When the application is fast enough to meet your users’ requirements.
  3. When there are no unnecessary calls and the calls that remain run at hardware speed.
There is some merit in all three answers, but as Dave Ensor taught me inside Oracle many years ago, the correct answer is C. Answer A specifically restricts your scope of concern to today, which is especially dangerous for developers. Answer B permits you to promote horrifically bad code, unhindered, into production, where it can hurt the performance of every function on the system. Answers&nnbsp;A and B both presume that you know information that you probably don’t know and that you may not need to know. Answer C is my favorite answer because it is tells you exactly when you’re done, using units you can measure and that you should be measuring.

Answer C is usually a tougher standard than answer A or B, and when it’s not, it is the best possible standard you can meet without upgrading or redesigning something. In light of this “tougher standard” kind of talk, it is still important to understand that what is optimal from a software engineering perspective is not always optimal from a business perspective. The term optimized must ultimately be judged within the constraints of what the business chooses to pay for. In the spirit of answer A, you can still make the decision not to optimize all your code to the last picosecond of its potential. How perfect you make your code should be a business decision. That decision should be informed by facts, and these facts should include knowledge of your code’s top speed.

Thank you, Guđmundur Jósepsson, of Iceland, for your question. Thank you for waiting patiently for several weeks while I struggled putting these thoughts into words.
06:33 What I Wanted to Tell Terry Bradshaw (2355 Bytes) » Cary Millsap
I met Terry Bradshaw one time. It was about ten years ago, in front of a movie theater near where I live.

When I was little, Terry Bradshaw was my enemy because, unforgivably to a young boy, he and his Pittsburgh Steelers kept beating my beloved Dallas Cowboys in Super Bowls. As I grew up, though, his personality on TV talk shows won me over, and I enjoy watching him to this day on Fox NFL Sunday. After learning a little bit about his life, I’ve grown to really admire and respect him.

I had heard that he owned a ranch not too far from where I live, and so I had it in mind that inevitably I would meet him someday, and I would say thank you. One day I had that chance.

I completely blew it.

My wife and I saw him there at the theater one day, standing by himself not far from us. It seemed like if I were to walk over and say hi, maybe it wouldn’t bother him. So I walked over, a little bit nervous. I shook his hand, and I said, “Mr. Bradshaw, hi, my name is Cary.” I would then say this:

I was a big Roger Staubach fan growing up. I watched Cowboys vs. Steelers like I was watching Good vs. Evil.

But as I’ve grown up, I have gained the deepest admiration and respect for you. You were a tremendous competitor, and you’re one of my favorite people to see on TV. Every time I see you, you bring a smile to my face. You’ve brought joy to a lot of people.

I just wanted to say thank you.

Yep, that’s what I would say to Terry Bradshaw if I got the chance. But that’s not how it would turn out. How it actually went was like this, …my big chance:

Me: I was a big Roger Staubach fan growing up.
TB: Hey, so was I!
Me: (stunned)
TB: (turns away)
The End

I was heartbroken. It bothers me still today. If you know Terry Bradshaw or someone who does, I wish you would please let him know. It would mean a lot to me.

…I did learn something that day about the elevator pitch.
06:33 Using Agile Practices to Create an Agile Presentation (4297 Bytes) » Cary Millsap
What’s the best way to make a presentation on Agile practices? Practice Agile practices.

You could write a presentation “big bang” style, delivering version 1.0 in front of your big audience of 200+ people at Kscope 2011 before anybody has seen it. Of course, if you do it that way, you build a lot of risk into your product. But what else can you do?

You can execute the Agile practices of releasing early and often, allowing the reception of your product to guide its design. Whenever you find an aspect of your product that doesn’t get the enthusiastic reception you had hoped for, you fix it for the next release.

That’s one of the reasons that my release schedule for “My Case for Agile Methods” includes a little online webinar hosted by Red Gate Software next week. My release schedule is actually a lot more complicated than just one little pre-ODTUG webinar:

2011-04-15Show key conceptual graphics to son (age 13)
2011-04-29Review #1 of paper with employee #1
2011-05-18Review #2 of paper with customer
2011-05-14Review #3 of paper with employee #1
2011-05-18Review #4 of paper with employee #2
2011-05-26Review #5 of paper with employee #3
2011-06-01Submit paper to ODTUG web site
2011-06-02Review #6 of paper with employee #1
2011-06-06Review #7 of paper with employee #3
2011-06-10Submit revised paper to ODTUG web site
2011-06-13Present “My Case for Agile Methods” to twelve people in an on-site customer meeting
2011-06-22Present “My Case for Agile Methods” in an online webinar hosted by Red Gate Software
2011-06-27Present “My Case for Agile Methods” at ODTUG Kscope 2011 in Long Beach, California

(By the way, the vast majority of the work here is done in Pages, not Keynote. I think using a word processor, not an operating system for slide projectors.)

Two Agile practices are key to everything I’ve ever done well: incremental design and rapid iteration. Release early, release often, and incorporate what you learn from real world use back into the product. The magic comes from learning how to choose wisely in two dimensions:
  1. Which feature do you include next?
  2. To whom do you release next?
The key is to show your work to other people. Yes, there’s tremendous value in practicing a presentation, but practicing without an audience merely reinforces, it doesn’t inform. What you need while you design something is information—specifically, you need the kind of information called feedback. Some of the feedback I receive generates some pretty energetic arguing. I need that to fortify my understanding of my own arguments so that I’ll be more likely to survive a good Q&A session on stage.

To lots of people who have seen teams run projects into the ground using what they call “Agile,” the word “Agile” has become a synonym for sloppy, irresponsible work habits. When you hear me talk about Agile, you’ll hear about practices that are highly disciplined and that actually require a lot of focus, dedication, commitment, practice, and plain old hard work to execute.

Agile, to me, is about injecting discipline into a process that is inevitably rife with unpredictable change.
06:33 The “Two Spaces After a Period” Thing (8336 Bytes) » Cary Millsap
Once upon a time, I told my friend Chet Justice why he should start using one space instead of two after a sentence-ending period. I’m glad I did.

Here’s the story.

When you type, you’re inputting data into a machine. I know you like feeling like you’re in charge, but really you’re not in charge of all the rules you have to follow while you’re inputting your data. Other people—like the designers of the machine you’re using—have made certain rules that you have to live by. For example, if you’re using a QWERTY keyboard, then the ‘A’ key is in a certain location on the keyboard, and whether it makes any sense to you or not, the ‘B’ key is way over there, not next to the ‘A’ key like you might have expected when you first started learning how to type. If you want a ‘B’ to appear in the input, then you have to reach over there and push the ‘B’ key on the keyboard.

In addition to the rules imposed upon you by the designers of the machine you’re using, you follow other rules, too. If you’re writing a computer program, then you have to follow the syntax rules of the language you’re using. There are alphabet and spelling and grammar rules for writing in German, and different ones for English. There are typographical rules for writing for The New Yorker, and different ones for the American Mathematical Society.

A lot of people who are over about 40 years old today learned to type on an actual typewriter. A typewriter is a machine that used rods and springs and other mechanical elements to press metal dies with backwards letter shapes engraved onto them through an inked ribbon onto a piece of paper. Some of the rules that governed the data input experience on typewriters included:
  • You had to learn where the keys were on the keyboard.
  • You had to learn how to physically return the carriage at the end of a line.
  • You had to learn your project’s rules of spelling.
  • You had to learn your project’s rules of grammar.
  • You had to learn your project’s rules of typography.
The first two rules listed here are physical, but the final three are syntactic and semantic. Just like you wouldn’t press the ‘A’ key to make a ‘B’, you wouldn’t use the strings “definately” or “we was” to make an English sentence.

On your typewriter, you might not have realized it, but you did adhere to some typography rules. They might have included:
  • Use two carriage returns after a paragraph.
  • Type two spaces after a sentence-ending period.
  • Type two spaces after a colon.
  • Use two consecutive hyphens to represent an em dash.
  • Make paragraphs no more than 80 characters wide.
  • Never use a carriage return between “Mr.” and the proper name that follows, or between a number and its unit.
The rules were different for different situations. For example, when I wrote a book back in the mid 1980s, one of the distinctive typography rules my publisher imposed upon me was:
  • Double-space all paragraph text.
They wanted their authors to do this so that their copyeditor had plenty of room for markup. Such typography rules can vary from one project to another.

Most people who didn’t write for different publishers got by just fine on the one set of typography rules they learned in high school. To them, it looked like there were only a few simple rules, and only one set of them. Most people had never even heard of a lot of the rules they should have been following, like rules about widows and orphans.

In the early 1980s, I began using computers for most of my work. I can remember learning how to use word processing programs like WordStar and Sprint. The rules were a lot more complicated with word processors. Now there were rules about “control keys” like ^X and ^Y, and there were no-break spaces and styles and leading and kerning and ligatures and all sorts of new things I had never had to think about before. A word processor was much more powerful than a typewriter. If you did it right, typesetting could could make your work look like a real book. But word processors revealed that typesetting was way more complicated than just typing.

Doing your own typesetting can be kind of like doing your own oil changes. Most people prefer to just put gas in the tank and not think too much about the esoteric features of their car (like their tires or their turn signal indicators). Most people who went from typewriters to word processors just wanted to type like they always had, using the good-old two or three rules of typography that had been long inserted into their brains by their high school teachers and then committed by decades of repetition.

Donald Knuth published The TeXBook in 1984. I think I bought it about ten minutes after it was published. Oh, I loved that book. Using TeX was my first real exposure to the world of actual professional-grade typography, and I have enjoyed thinking about typography ever since. I practice typography every day that I use Keynote or Pages or InDesign to do my work.

Many people don’t realize it, but when you type input into programs like Microsoft Word should follow typography rules including these:
  • Never enter a blank line (edit your paragraph’s style to manipulate its spacing).
  • Use a single space after a sentence-ending period (the typesetter software you’re using will make the amount of space look right as it composes the paragraph).
  • Use a non-breaking space after a non-sentence-ending period (so the typesetter software won’t break “Mr. Harkey” across lines).
  • Use a non-breaking space between a number and its unit (so the typesetter software won’t break “8 oz” across lines).
  • Use an en dash—not a hyphen—to specify ranges of numbers (like “3–8”).
  • Use an em dash—not a pair of hyphens—when you need an em dash (like in this sentence).
  • Use proper quotation marks, like “this” and ‘this’ (or even « this »).
Of course, you can choose to not follow these rules, just like you can choose to be willfully ignorant about spelling or grammar. But to a reader who has studied typography even just a little bit, seeing you break these rules feels the same as seeing a sentence like, “You was suppose to use apostrophe's.” It affects how people perceive you.

So, it’s always funny to me when people get into heated arguments on Facebook about using one space or two after a period. It’s the tiniest little tip of the typography iceberg, but it opens the conversation about typography, for which I’m glad. In these discussions, two questions come up repeatedly: “When did the rule change? Why?”

Well, the rule never did change. The next time I type on an actual typewriter, I will use two spaces after each sentence-ending period. I will also use two spaces when I create a Courier font court document or something that I want to look like it was created in the 1930s. But when I work on my book in Adobe InDesign, I’ll use one space. When I use my iPhone, I’ll tap in two spaces at the end of a sentence, because it automatically replaces them with a period and a single space. I adapt to the rules that govern the situation I’m in.

It’s not that the rules have changed. It’s that the set of rules was always a lot bigger than most people ever knew.
06:33 The String Puzzle (9391 Bytes) » Cary Millsap
I gave my two boys an old puzzle to solve yesterday. I told them that I’d give them each $10 if they could solve it for me. It’s one of the ways we do the “allowance” thing around the house sometimes.

Here’s the puzzle. A piece of string is stretched tightly around the Earth along its equator. Imagine that this string along the equator forms a perfect circle, and imagine that to reach around that perfect circle, the string has to be exactly 25,000 miles long. Now imagine that you wanted to suspend this string 4 inches above the surface of the Earth, all the way around it. How much longer would the string have to be do do this?


Before you read any further, guess the answer. How much longer would the string have to be? A few inches? Several miles? What do you think?

Now, my older son Alex was more interested in the problem than I thought he would be. He knows the formula for computing the circumference of a circle as a function of its diameter, and he knew that raising the string 4 inches above the surface constituted a diameter change. So the kernel of a solution had begun to formulate in his head. And he had a calculator handy, which he loves to use.

We were at Chipotle for dinner. The rest of the family went in to order, and Alex waited in the truck to solve the problem “where he could have some peace and quiet.” He came into the restaurant in time to order, and he gave me a number that he had cooked up on his calculator in the truck. I had no idea whether it was correct or not (I haven’t worked the problem in many years), so I told him to explain to me how he got it.

When he explained to me what he had done, he pretty quickly discovered that he had made a unit conversion error. He had manipulated the ‘25,000’ and the ‘4’ as if they had been expressed in the same units, so his answer was wrong, but it sounded like conceptually he got what he needed to do to solve the problem. So I had him write it down. On a napkin, of course:


The first thing he did was draw a sphere (top center) and tell me that the diameter of this sphere is 25,000 miles divided by 3.14 (the approximation of π that they use at school). He started dividing that out on his calculator when I pulled the “Whoa, wait” thing where I asked him why he was dividing those two quantities, which caused him, grudgingly, to write out that C = 25,000 mi, that C = πd, and that therefore d = C/π. So I let him figure out that d ≈ 7,961 mi. There’s loss of precision there, because of the 3.14 approximation, and because there are lots of digits to the right of the decimal point after ‘7961’, but more about that later.

I told him to call the length of the original string C (for circumference) and to call the 4-inch suspension distance of the string h (for height), and then write me the formula for the length of the 4-inch high string, without worrying about any unit conversion issues. He got the formula pretty close on the first shot. He added 4 inches to the diameter of the circle instead of adding 4 inches to the radius (you can see the ‘4’ scratched out and replaced with an ‘8’ in the “8 in/63360 in” expression in the middle of the napkin. Where did the ‘63360’ come from, I asked? He explained that this is the number of inches in a mile (5,280 × 12). Good.

But I asked him to hold off on the unit conversion stuff until the very end. He wrote the correct formula for the length of the new string, which is [(C/π) + 2h]·π (bottom left). Then I let him run the formula out on his calculator. It came out to something bigger than exactly 25,000; I didn’t even look at what he got. This number he had produced minus 25,000 would be the answer we were looking for, but I knew there would be at least two problems with getting the answer this way:
  • The value of π is approximately 3.14, but it’s not exactly 3.14.
  • Whenever he had to transfer a precise number from one calculation to the next, I knew Alex was either rounding or truncating liberally.
So, I told him we were going to work this problem out completely symbolically, and only plug the numbers in at the very end. It turns out that doing the problem this way yields a very nice little surprise.

Here’s my half of the napkin:


I called the new string length cʹ and the old string length c. The answer to the puzzle is the value of cʹ − c.

The new circumference cʹ will be π times the new diameter, which is c/π + 2h, as Alex figured out. The second step distributes the π factor through the addition, resulting in cʹ − c = πc/π + 2πh − c. The πc/π term simplifies to just c, and it’s the final step where the magic happens: cʹ − c = c + 2πhc reduces simply to cʹ − c = 2πh. The difference between the new string length and the old one is 2πh, which in our case (where h = 4 inches) is roughly 25.133 inches.

So, problem solved. The string will have to be about 25.133 inches longer if we want to suspend it 4 inches above the surface.

Notice how simple the solution is: the only error we have to worry about is how precisely we want to represent π in our calculation.

Here’s the even cooler part, though: there is no ‘c’ in the formula for the answer. Did you notice that? What does that mean?

It means that the original circumference doesn’t matter. It means that if we have a string around the Moon that we want to raise 4 inches off the surface, we just need another 25.133 inches. How about a string stretched around Jupiter? just 25.133 more inches. Betelgeuse, a star whose diameter is about the same size as Jupiter’s orbit? Just 25.133 more inches. The whole solar system? Just 25.133 more inches. The entire Milky Way galaxy? Just 25.133 more inches. A golf ball? Again, 25.133 more inches. A single electron? Still 25.133 inches.

This is the kind of insight that solving a problem symbolically provides. A numerical solution tends to answer a question and halt the conversation. A symbolic formula answers our question and invites us to ask more.

The calculator answer is just a fish (pardon the analogy, but a potentially tainted fish at that). The symbolic answer is a fishing pole with a stock pond.

So, did I pay Alex for his answer? No. Giving two or three different answers doesn’t close the deal, even if one of the answers is correct. He doesn’t get paid for blurting out possible answers. He doesn’t even get paid for answering the question correctly; he gets paid for convincing me that he has created a correct answer. In the professional world, that is the key: the convincing.

Imagine that a consultant or a salesman told you that you needed to execute a $250,000 procedure to make your computer application run faster. Would you do it? Under what circumstances? If you just trusted him and did it, but it didn’t do what you had hoped, would you ever trust him again? I would argue that you shouldn’t trust an answer without a compelling rationale, and that the recommender’s reputation alone is not a compelling rationale.

The deal is, whenever Alex can show me the right answer and convince me that he’s done the problem correctly, that’s when I’ll give him the $10. I’m guessing it’ll happen within the next three days or so. The interesting bet is going to be whether his little brother beats him to it.
06:33 The Fundamental Challenge of Computer System Performance (12060 Bytes) » Cary Millsap
The fundamental challenge of computer system performance is for your system to have enough power to handle the work you ask it to do. It sounds really simple, but helping people meet this challenge has been the point of my whole career. It has kept me busy for 26 years, and there’s no end in sight.

Capacity and Workload

Our challenge is the relationship between a computer’s capacity and its workload. I think of capacity as an empty box representing a machine’s ability to do work over time. Workload is the work your computer does, in the form of programs that it runs for you, executed over time. Workload is the content that can fill the capacity box.


Capacity Is the One You Can Control, Right?

When the workload gets too close to filling the box, what do you do? Most people’s instinctive reaction is that, well, we need a bigger box. Slow system? Just add power. It sounds so simple, especially since—as “everyone knows”—computers get faster and cheaper every year. We call that the KIWI response: kill it with iron.

KIWI... Why Not?

As welcome as KIWI may feel, KIWI is expensive, and it doesn’t always work. Maybe you don’t have the budget right now to upgrade to a new machine. Upgrades cost more than just the hardware itself: there’s the time and money it takes to set it up, test it, and migrate your applications to it. Your software may cost more to run on faster hardware. What if your system is already the biggest and fastest one they make?

And as weird as it may sound, upgrading to a more powerful computer doesn’t always make your programs run faster. There are classes of performance problems that adding capacity never solves. (Yes, it is possible to predict when that will happen.) KIWI is not always a viable answer.

So, What Can You Do?

Performance is not just about capacity. Though many people overlook them, there are solutions on the workload side of the ledger, too. What if you could make workload smaller without compromising the value of your system?
It is usually possible to make a computer produce all of the useful results that you need without having to do as much work.
You might be able to make a system run faster by making its capacity box bigger. But you might also make it run faster by trimming down that big red workload inside your existing box. If you only trim off the wasteful stuff, then nobody gets hurt, and you’ll have winning all around.

So, how might one go about doing that?

Workload

“Workload” is a conjunction of two words. It is useful to think about those two words separately.


The amount of work your system does for a given program execution is determined mostly by how that program is written. A lot of programs make their systems do more work than they should. Your load, on the other hand—the number of program executions people request—is determined mostly by your users. Users can waste system capacity, too; for example, by running reports that nobody ever reads.

Both work and load are variables that, with skill, you can manipulate to your benefit. You do it by improving the code in your programs (reducing work), or by improving your business processes (reducing load). I like workload optimizations because they usually save money and work better than capacity increases. Workload optimization can seem like magic.

The Anatomy of Performance

This simple equation explains why a program consumes the time it does:
r = cl        or        response time = call count × call latency
Think of a call as a computer instruction. Call count, then, is the number of instructions that your system executes when you run a program, and call latency is how long each instruction takes. How long you wait for your answer, then—your response time—is the product of your call count and your call latency.

Some fine print: It’s really a little more complicated than this, but actually not that much. Most response times are composed of many different types of calls, all of which have different latencies (we see these in program execution profiles), so the real equation looks like r = c1l1 + c2l2 + ... + cnln. But we’ll be fine with r = cl for this article.

Call count depends on two things: how the code is written, and how often people run that code.
  • How the code is written (work) — If you were programming a robot to shop for you at the grocery store, you could program it to make one trip from home for each item you purchase. Go get bacon. Come home. Go get milk... It would probably be dumb if you did it that way, because the duration of your shopping experience would be dominated by the execution of clearly unnecessary travel instructions, but you’d be surprised at how often people write programs that act like this.
  • How often people run that code (load) — If you wanted your grocery store robot to buy 42 things for you, it would have to execute more instructions than if you wanted to buy only 7. If you found yourself repeatedly discarding spoiled, unused food, you might be able to reduce the number of things you shop for without compromising anything you really need.
Call latency is influenced by two types of delays: queueing delays and coherency delays.
  • Queueing delays — Whenever you request a resource that is already busy servicing other requests, you wait in line. That’s a queueing delay. It’s what happens when your robot tries to drive to the grocery store, but all the roads are clogged with robots that are going to the store to buy one item at a time. Driving to the store takes only 7 minutes, but waiting in traffic costs you another 13 minutes. The more work your robot does, the greater its chances of being delayed by queueing, and the more such delays your robot will inflict upon others as well.
  • Coherency delays — You endure a coherency delay whenever a resource you are using needs to communicate or coordinate with another resource. For example, if your robot’s cashier at the store has to talk with a specific manager or other cashier (who might already be busy with a customer), the checkout process will take longer. The more times your robot goes to the store, the worse your wait will be, and everyone else’s, too.

The Secret

This r = cl thing sure looks like the equation for a line, but because of queueing and coherency delays, the value of l increases when c increases. This causes response time to act not like a line, but instead like a hyperbola.


Because our brains tend to conceive of our world as linear, nobody expects for everyone’s response times to get seven times worse when you’ve only added some new little bit of workload, but that’s the kind of thing that routinely happens with performance. ...And not just computer performance. Banks, highways, restaurants, amusement parks, and grocery-shopping robots all work the same way.

Response times are trememdously sensitive to your call counts, so the secret to great performance is to keep your call counts small. This principle is the basis for perhaps the best and most famous performance optimization advice ever rendered:
The First Rule of Program Optimization: Don’t do it.

The Second Rule of Program Optimization (for experts only!): Don’t do it yet.

The Problem

Keeping call counts small is really, really important. This makes being a vendor of information services difficult, because it is so easy for application users to make call counts grow. They can do it by running more programs, by adding more users, by adding new features or reports, or by even by just the routine process of adding more data every day.

Running your application with other applications on the same computer complicates the problem. What happens when all these application’ peak workloads overlap? It is a problem that Application Service Providers (ASPs), Software as a Service (SaaS) providers, and cloud computing providers must solve.

The Solution

The solution is a process:
  1. Call counts are sacred. They can be difficult to forecast, so you have to measure them continually. Understand that. Hire people who understand it. Hire people who know how to measure and improve the efficiency of your application programs and the systems they reside on.
  2. Give your people time to fix inefficiencies in your code. An inexpensive code fix might return many times the benefit of an expensive hardware upgrade. If you have bought your software from a software vendor, work with them to make sure they are streamlining the code they ship you.
  3. Learn when to say no. Don’t add new features (especially new long-running programs like reports) that are inefficient, that make more calls than necessary. If your users are already creating as much workload as the system can handle, then start prioritizing which workload you will and won’t allow on your system during peak hours.
  4. If you are an information service provider, charge your customers for the amount of work your systems do for them. The economic incentive to build and buy more efficient programs works wonders.
06:33 NoSQL and Oracle, Sex and Marriage (8301 Bytes) » Cary Millsap
At last week’s Dallas Oracle Users Group meeting, an Oracle DBA asked me, “With all the new database alternatives out there today, like all these open source NoSQL databases, would you recommend for us to learn some of those?”

I told him I had a theory about how these got so popular and that I wanted to share that before I answered his question.

My theory is this. Developers perceive Oracle as being too costly, time-consuming, and complex:
  • An Oracle Database costs a lot. If you don’t already have an Oracle license, it’s going to take time and money to get one. On the other hand, you can just install Mongo DB today.
  • Even if you have an Oracle site-wide license, the Oracle software is probably centrally controlled. To get an installation done, you’re probably going to have to negotiate, justify, write a proposal, fill out forms, ...you know, supplicate yourself to—er, I mean negotiate with—your internal IT guys to get an Oracle Database installed. It’s a lot easier to just install MySQL yourself.
  • Oracle is too complicated. Even if you have a site license and someone who’s happy to install it for you, it’s so big and complicated and proprietary... The only way to run an Oracle Database is with SQL (a declarative language that is alien to many developers) executed through a thick, proprietary, possibly even difficult-to-install layer of software like Oracle Enterprise Manager, Oracle SQL Developer, or sqlplus. Isn’t there an open source database out there that you could just manage from your operating system command line?
When a developer is thinking about installing a database today because he need one to write his next feature, he wants something cheap, quick, and lightweight. None of those constraints really sounds like Oracle, does it?

So your Java developers install this NoSQL thing, because it’s easy, and then they write a bunch of application code on top of it. Maybe so much code that there’s no affordable way to turn back. Eventually, though, someone will accidentally crash a machine in the middle of something, and there’ll be a whole bunch of partway finished jobs that die. Out of all the rows that are supposed to be in the database, some will be there and some won’t, and so now your company will have to figure out how to delete the parts of those jobs that aren’t supposed to be there.

Because now everyone understands that this kind of thing will probably happen again, too, the exercise may well turn into a feature specification for various “eraser” functions for the application, which (I hope, anyway) will eventually lead to the team discovering the technical term transaction. A transaction is a unit of work that must be atomic, consistent, isolated, and durable (that’where this acronym ACID comes from). If your database doesn’t guarantee that every arbitrarily complex unit of work (every transaction) makes it either 100% into the database or not at all, then your developers have to write that feature themselves. That’s a big, tremendously complex feature. On an Oracle Database, the transaction is a fundamental right given automatically to every user on the system.

Let’s look at just that ‘I’ in ACID for a moment: isolation. How big a deal is transaction isolation? Imagine that your system has a query that runs from 1 pm to 2 pm. Imagine that it prints results to paper as it runs. Now suppose that at 1:30 pm, some user on the system updates two rows in your query’s base table: the table’s first row and its last row. At 1:30, the pre-update version of that first row has already been printed to paper (that happened shortly after 1 pm). The question is, what’s supposed to happen at 2 pm when it comes time to print the information for the final row? You should hope for the old value of that final row—the value as of 1 pm—to print out; otherwise, your report details won’t add up to your report totals. However, if your database doesn’t handle that transaction isolation feature for you automatically, then either you’ll have to lock the table when you run the report (creating an 30-minute-long performance problem for the person wanting to update the table at 1:30), or your query will have to make a snapshot of the table at 1 pm, which is going to require both a lot of extra code and that same lock I just described. On an Oracle Database, high-performance, non-locking read consistency is a standard feature.

And what about backups? Backups are intimately related to the read consistency problem, because backups are just really long queries that get persisted to some secondary storage device. Are you going to quiesce your whole database—freeze the whole system—for whatever duration is required to take a cold backup? That’s the simplest sounding approach, but if you’re going to try to run an actual business with this system, then shutting it down every day—taking down time—to back it up is a real operational problem. Anything fancier (for example, rolling downtime, quiescing parts of your database but not the whole thing) will add cost, time, and complexity. On an Oracle Database, high-performance online “hot” backups are a standard feature.

The thing is, your developers could write code to do transactions (read consistency and all) and incremental (“hot”) backups. Of course they could. Oh, and constraints, and triggers (don’t forget to remind them to handle the mutating table problem), and automatic query optimization, and more, ...but to write those features Really Really Well™, it would take them 30 years and a hundred of their smartest friends to help write it, test it, and fund it. Maybe that’s an exaggeration. Maybe it would take them only a couple years. But Oracle has already done all that for you, and they offer it at a cost that doesn’t seem as high once you understand what all is in there. (And of course, if you buy it on May 31, they’ll cut you a break.)

So I looked at the guy who asked me the question, and I told him, it’s kind of like getting married. When you think about getting married, you’re probably focused mostly on the sex. You’re probably not spending too much time thinking, “Oh, baby, this is the woman I want to be doing family budgets with in fifteen years.” But you need to be. You need to be thinking about the boring stuff like transactions and read consistency and backups and constraints and triggers and automatic query optimization when you select the database you’re going to marry.

Of course, my 15-year-old son was in the room when I said this. I think he probably took it the right way.

So my answer to the original question—“Should I learn some of these other technologies?”—is “Yes, absolutely,” for at least three reasons:
  • Maybe some development group down the hall is thinking of installing Mongo DB this week so they can get their next set of features implemented. If you know something about both Mongo DB and Oracle, you can help that development group and your managers make better informed decisions about that choice. Maybe Mongo DB is all they need. Maybe it’s not. You can help.
  • You’re going to learn a lot more than you expect when you learn another database technology, just like learning another natural language (like English, Spanish, etc.) teaches you things you didn’t expect to learn about your native language.
  • Finally, I encourage you to diversify your knowledge, if for no other reason than your own self-confidence. What if market factors conspire in such a manner that you find yourself competing for an Oracle-unrelated job? A track record of having learned at least two database technologies is proof to yourself that you’re not going to have that much of a problem learning your third.
06:33 My Friend Karen (3000 Bytes) » Cary Millsap
My friend Karen Morton passed away on July 23, 2015 after a four-month battle against cancer. You can hear her voice here.

I met Karen Morton in February 2002. The day I met her, I knew she was awesome. She told me the story that, as a consultant, she had been doing something that was unheard-of. She guaranteed her clients that if she couldn’t make things on their systems go at least X much faster on her very first day, then they wouldn’t have to pay. She was a Give First person, even in her business. That is really hard to do. After she told me this story, I asked the obvious question. She smiled her big smile and told me that her clients had always paid her—cheerfully.

It was an honor when Karen joined my company just a little while later. She was the best teammate ever, and she delighted every customer she ever met. The times I got to work with Karen were bright spots in my life, during many of the most difficult years of my career. For me, she was a continual source of knowledge, inspiration, and courage.

This next part is for Karen’s family and friends outside of work. You know that she was smart, and you know she was successful. What you may not realize is how successful she was. Your girl was famous all over the world. She was literally one of the top experts on Earth at making computing systems run faster. She used her brilliant gift for explaining things through stories to become one of the most interesting and fun presenters in the Oracle world to go watch, and her attendance numbers proved it. Thousands of people all over the world know the name, the voice, and the face of your friend, your daughter, your sister, your spouse, your mom.

Everyone loved Karen’s stories. She and I told stories and talked about stories, it seems like, all the time we were together. Stories about how Oracle works, stories about helping people, stories about her college basketball career, stories about our kids and their sports, ...

My favorite stories of all—and my family’s too—were the stories about her younger brother Ted. These stories always started out with some middle-of-the-night phone call that Karen would describe in her most somber voice, with the Tennessee accent turned on full-bore: “Kar’n: This is your brother, Theodore LeROY.” Ted was Karen’s brother Teddy Lee when he wasn’t in trouble, so of course he was always Theodore LeROY in her stories. Every story Karen told was funny and kind.

We all wanted to have more time with Karen than we got, but she touched and warmed the lives of literally thousands of people. Karen Morton used her half-century here on Earth with us as well as anyone I’ve ever met. She did it right.

God bless you, Karen. I love you.
06:33 Messed-Up App of the Day: Tables of Numbers (5988 Bytes) » Cary Millsap
Quick, which database is the biggest space consumer on this system?
Database                  Total Size   Total Storage
-------------------- --------------- ---------------
SAD99PS 635.53 GB 1.24 TB
ANGLL 9.15 TB 18.3 TB
FRI_W1 2.14 TB 4.29 TB
DEMO 6.62 TB 13.24 TB
H111D16 7.81 TB 15.63 TB
HAANT 1.1 TB 2.2 TB
FSU 7.41 TB 14.81 TB
BYNANK 2.69 TB 5.38 TB
HDMI7 237.68 GB 476.12 GB
SXXZPP 598.49 GB 1.17 TB
TPAA 1.71 TB 3.43 TB
MAISTERS 823.96 GB 1.61 TB
p17gv_data01.dbf 800.0 GB 1.56 TB
It’s harder than it looks.

Did you come up with ANGLL? If you didn’t, then you should look again. If you did, then what steps did you have to execute to find the answer?

I’m guessing you did something like I did:
  1. Skim the entire list. Notice that HDMI7 has a really big value in the third column.
  2. Read the column headings. Parse the difference in meaning between “size” and “storage.” Realize that the “storage” column is where the answer to a question about space consumption will lie.
  3. Skim the “Total Storage” column again and notice that the wide “476.12” number I found previously has a GB label beside it, while all the other labels are TB.
  4. Skim the table again to make sure there’s no PB in there.
  5. Do a little arithmetic in my head to realize that a TB is 1000× bigger than a GB, so 476.12 is probably not the biggest number after all, in spite of how big it looked.
  6. Re-skim the “Total Storage” column looking for big TB numbers.
  7. The biggest-looking TB number is 15.63 on the H111D16 row.
  8. Notice the trap on the ANGLL row that there are only three significant digits showing in the “18.3” figure, which looks physically the same size as the three-digit figures “1.24” and “4.29” directly above and below it, but realize that 18.3 (which should have been rendered “18.30”) is an order of magnitude larger.
  9. Skim the column again to make sure I’m not missing another such number.
  10. The answer is ANGLL.
That’s a lot of work. Every reader who uses this table to answer that question has to do it.

Rendering the table differently makes your readers’ (plural!) job much easier:
Database          Size (TB)  Storage (TB)
---------------- --------- ------------
SAD99PS .64 1.24
ANGLL 9.15 18.30
FRI_W1 2.14 4.29
DEMO 6.62 13.24
H111D16 7.81 15.63
HAANT 1.10 2.20
FSU 7.41 14.81
BYNANK 2.69 5.38
HDMI7 .24 .48
SXXZPP .60 1.17
TPAA 1.71 3.43
MAISTERS .82 1.61
p17gv_data01.dbf .80 1.56
This table obeys an important design principle:
The amount of ink it takes to render each number is proportional to its relative magnitude.
I fixed two problems: (i) now all the units are consistent (I have guaranteed this feature by adding unit label to the header and deleting all labels from the rows); and (ii) I’m showing the same number of significant digits for each number. Now, you don’t have to do arithmetic in your head, and now you can see more easily that the answer is ANGLL, at 18.30 TB.

Let’s go one step further and finish the deal. If you really want to make it as easy as possible for readers to understand your space consumption problem, then you should sort the data, too:
Database          Size (TB)  Storage (TB)
---------------- --------- ------------
ANGLL 9.15 18.30
H111D16 7.81 15.63
FSU 7.41 14.81
DEMO 6.62 13.24
BYNANK 2.69 5.38
FRI_W1 2.14 4.29
TPAA 1.71 3.43
HAANT 1.10 2.20
MAISTERS .82 1.61
p17gv_data01.dbf .80 1.56
SAD99PS .64 1.24
SXXZPP .60 1.17
HDMI7 .24 .48
Now, your answer comes in a glance. Think back at the comprehension steps that I described above. With the table here, you only need:
  1. Notice that the table is sorted in descending numerical order.
  2. Comprehend the column headings.
  3. The answer is ANGLL.
As a reader, you have executed far less code path in your brain to completely comprehend the data that the author wants you to understand.

Good design is a topic of consideration. And even conservation. If spending 10 extra minutes formatting your data better saves 1,000 readers 2 minutes each, then you’ve saved the world 1,990 minutes of wasted effort.

But good design is also a very practical matter for you personally, too. If you want your audience to understand your work, then make your information easier for them to consume—whether you’re writing email, proposals, reports, infographics, slides, or software. It’s part of the pathway to being more persuasive.

06:33 Messed-Up App of the Day: Crux CCH-01W (8854 Bytes) » Cary Millsap
Today’s Messed-Up App of the Day is the “Crux CCH-01W rear-view camera for select 2007-up Jeep Wrangler models.”

A rear-view camera is an especially good idea in the Jeep Wrangler, because it is very difficult to see behind the vehicle. The rear seat headrests, the wiper motor housing, the spare tire, and the center brake light all conspire to obstruct much of what little view the window had given you to begin with.

The view is so bad that it’s easy to, for example, accidentally demolish a mailbox.

I chose the Crux CCH-01W because it is purpose-built for our 2012 Jeep Wrangler. It snaps right into the license plate frame. I liked that. It had 4.5 out of 5.0 stars in four reviews at crutchfield.com, my favorite place to buy stuff like this. I liked that, too.

But I do not like the Crux CCH-01W. I returned it because our Jeep will be safer without this camera than with it. Here’s the story.

My installation process was probably pretty normal. I had never done a project like this before, so it took me longer than it should have. Crux doesn’t include any installation instructions with the camera, which is a little frustrating, but I knew that from the reviews. There is a lot of help online, and Crutchfield helped as much as I needed. After all the work of installing it, it was a huge thrill when I first shifted into Reverse and—voilà!—a picture appeared in my dashboard.

However, that was where the happiness would end. When I tried to use the camera, I noticed right away that the red, yellow, and green grid lines that the camera superimposes upon its picture didn’t make any sense. The grid lines showed that I was going to collide with the vehicle on my left that clearly wasn’t in jeopardy (an inconvenient false negative), and they showed that I was all-clear on the right when in fact I was about to ram into my garage door facing (a dangerous false positive).

The problem is that the grid lines are offset about two feet to the left. Of course, this is because the camera is about two feet to the left of the vehicle’s centerline. It’s above the license plate, below the left-hand tail light.

So then, to use these grid lines, you have to shift them in your mind about two feet to the right. In your mind. There’s no way to adjust them on the screen. Since this camera is designed exclusively for the left-hand corner of a 2007-up Jeep Wrangler, shouldn’t the designers have adjusted the location of the grid lines to compensate?

So, let’s recap. The safety device I bought to relieve driver workload and improve safety will, unfortunately, increase driver workload and degrade safety.

That’s bad enough, but it doesn’t end there. There is a far worse problem than just the misalignment of the grid lines.

Here is a photo of a my little girl standing a few feet behind the Jeep, directly behind the right rear wheel:

And here is what the camera shows the driver while she is standing there:

No way am I keeping that camera on the vehicle.

It’s easy to understand why it happens. The camera, which has a 120° viewing angle, is located so far off the vehicle centerline that it creates a blind spot behind the right-hand corner of the vehicle and grid lines that don’t make sense.

The Crux CCH-01W is one of those products that seems like nobody who designed it ever actually had to use it. I think it should never have been released.

As I was shopping for this project, my son and a local professional installer advised me to buy a camera that mounted on the vehicle centerline instead of this one. I didn’t take their advice because the reviews for the CCH-01W were good, and the price was $170 less. Fortunately, Crutchfield has a generous return policy, and the center-mounting 170°-view replacement camera that I’ll install this weekend has arrived today.

I’ve learned a lot. The second installation will go much more quickly than the first.

06:33 Loss Aversion and the Setting of DB_BLOCK_CHECKSUM (4730 Bytes) » Cary Millsap
Within Accenture Enkitec Group, we have recently been discussing the Oracle db_block_checksum parameter and how difficult it is to get clients to set it to a safer setting.

Clients are always concerned about the performance impact of features like this. Several years ago, I met a lot of people who had—in response to some expensive advice with which I strongly disagreed—turned off redo logging with an underscore parameter. The performance they would get from doing this would set the expectation level in their mind, which would cause them to resist (strenuously!) any notion of switching this [now horribly expensive] logging back on. Of course, it makes you wish that it had never even been a parameter.

I believe that the right analysis is to think clearly about risk. Risk is a non-technical word in most people’s minds, but in finance courses they teach that risk is quantifiable as a probability distribution. For example, you can calculate the probability that a disk will go bad in your system today. For disks, it’s not too difficult, because vendors do those calculations (MTTF) for us. But the probability that you’ll wish you had set db_block_checksum=full yesterday is probably more difficult to compute.

From a psychology perspective, customers would be happier if their systems had db_block_checksum set to full or typical to begin with. Then in response to the question,
“Would you like to remove your safety net in exchange for going between 1% and 10% faster? Here’s the horror you might face if you do it...”
...I’d wager that most people would say no, thank you. They will react emotionally to the idea of their safety net being taken away.

But with the baseline of its being turned off to begin with, the question is,
“Would you like to install a safety net in exchange for slowing your system down between 1% and 10%? Here’s the horror you might face if you don’t...”
...I’d wager that most people would answer no, thank you, even though this verdict that is opposite to the one I predicted above. They will react emotionally to the idea of their performance being taken away.

Most people have a strong propensity toward loss aversion. They tend to prefer avoiding losses over acquiring gains. If they already have a safety net, they won’t want to lose it. If they don’t have the safety net they need, they’ll feel averse to losing performance to get one. It ends up being a problem more about psychology than technology.

The only tools I know to help people make the right decision are:
  1. Talk to good salespeople about how they overcome the psychology issue. They have to deal with it every day.
  2. Give concrete evidence. Compute the probabilities. Tell the stories of how bad it is to have insufficient protection. Explain that any software feature that provides a benefit is going to cost some system capacity (just like a new report, for example), and that this safety feature is worth the cost. Make sure that when you size systems, you include the incremental capacity cost of switching to db_block_checksum=full.
My teammates get it, of course, because they’ve lived the stories, over and over again, in their roles on the corruption team at Oracle Support. You can get it, too, without leaving your keyboard. If you want to see a fantastic and absolutely horrifying short story about what happens if you do not use Oracle’s db_block_checksum feature properly, read David Loinaz’s article now.

When you read David’s article, you are going to see heavy quoting of my post here in his intro. He did that with my full support. (He wrote his article when my article here wasn’t an article yet.) If you feel like you’ve read it before, just keep reading. You really, really need to see what David has written, beginning with the question:
If I’ve never faced a corruption, and I have good backup strategy, my disks are mirrored, and I have a great database backup strategy, then why do I need to set these kinds of parameters that will impact my performance?
Enjoy.
06:33 Is “Can’t” Really the Word You Want? (8610 Bytes) » Cary Millsap
My friend Chester (Chet Justice in real life; oraclenerd when he puts his cape on) tweeted this yesterday:
can’t lives on won’t street - i’m sure my son will hate me when he’s older for saying that all the time.

I like it. It reminds me of an article that I drafted a few months ago but hadn’t posted yet. Here it is...

When I ask for help sometimes, I find myself writing a sentence that ends with, “…but I can’t figure it out.” I try to catch myself when I do that, because can’t is not the correct word.

Here’s what can’t means. Imagine a line representing time, with the middle marking where you are at some “right now” instant in time. The leftward direction represents the past, and the rightward direction represents the future.


Can’t means that I am incapable of doing something at every point along this timeline: past, present, and future.


Now, of course, can’t is different from mustn’t—“must not”which means that you’re not supposed to try to do something, presumably because it’s bad for you. So I’m not talking about the can/may distinction that grammarians bring to your attention when you say, “Can I have a candy bar?” and then they say, “I don’t know, can you?” And then you have to say, “Ok, may I have a candy bar” to actually have candy bar. I digress.

Back to the timeline. There are other words you can use to describe specific parts of that timeline, and here is where it becomes more apparent that can’t is often just not the right word:
  • Didn’t, a contraction of “did not.” It means that you did not do something in the past. It doesn’t necessarily mean you couldn’t have, or that you were incapable of doing it; it’s just a simple statement of fact that you, in fact, did not.
  • Aren’t, a contraction of “are not.” It means that you are in fact not doing something right now. This is different from “don’t,” which often is used to state an intention about the future, too, as in “I don’t smoke,” or “I do not like them, Sam-I-am.”
  • Won’t, a contraction of “will not.” It means that you will not do something in the future. It doesn’t necessarily mean you couldn’t, or that you are going to be incapable of doing it; it’s just a simple statement of intention that you, right now, don’t plan to. That’s the funny thing about your future. Sometimes you decide to change your plans. That’s ok, but it means that sometimes when you say won’t, you’re going to be wrong.
Here’s how it all looks on the timeline:


So, when I ask for help and I almost say, “I can’t figure it out,” the truth is really only that “I didn’t figure it out.”

“...Yet.”

...Because, you see, I do not have complete knowledge about the future, so it is not correct for me to say that I will never figure it out. Maybe I will. However, I do have complete knowledge of the past (this one aspect of the past, anyway), and so it is correct to speak about that by saying that I didn’t figure it out, or that I haven’t figured it out.

Does it seem like I’m going through a lot of bother making such detailed distinctions about such simple words? It matters, though. The people around you are affected by what you say and write. Furthermore, you are affected by the the stuff you say and write. Not only do our thoughts affect our words, our words affect our thoughts. The way you say and write stuff changes how you think about stuff. So if you aspire to tell the truth (is that still a thing?), or if you just want to know more about the truth, then it’s important to get your words right.

Now, back to the timeline. Just because you haven’t done something doesn’t mean you can’t, which means that you never will. Can’t is an as-if-factual statement about your future. Careless use of the word can’t can hurt people when you say it about them. It can hurt you when you think it about yourself.

Listen for it; you’ll hear it all the time:
  • “Why can’t you sit still?!” If you ask the question more accurately, it kind of answers itself: “Why haven’t you sat still for the past hour and a half?” Well, dad, maybe it’s because I’m bored and, oh, maybe it’s because I’m five! Asking why you haven’t been sitting still reminds me that perhaps there’s something I can do to make it easier for both of us to work through the next five minutes, and that maybe asking you to sit still for the next whole hour is asking too much.
  • “You can’t run that fast.” Prove it. Just because you haven’t doesn’t mean you never will. Before 1954, people used to say you can’t run a four-minute mile, that nobody can. Today, over a thousand people have done it. Can you become the most commercially successful and critically acclaimed act in history of popular music if you can’t read music? Well, apparently you can: that’s what the Beatles did. I’m probably not going to, but understanding that it’s not impossible is more life-enriching than believing I can’t.
  • “You can’t be trusted.” Rough waters here, mate. If you’ve behaved in such a manner that someone would say this about you, then you’ve got a lot of work in front of you. But no. No matter what, so far, all you’ve demonstrated is that you have been untrustworthy in the past. A true statement about your past is not necessarily a true statement about your future. It’s all about the choices you make from this moment onward.
Lots of parents and teachers don’t like can’t for its de-motivational qualities. I agree: when you think you can’t, you most likely won’t, because you won’t even try. It’s Chet’s “WONT STREET”.

When you think clearly about its technical meaning, you can also see that it’s also a word that’s often just not true. I hate being wrong. So I try not to use the word can’t very often.
06:33 I Wish I Sold More (4779 Bytes) » Cary Millsap
I flew home yesterday from Karen’s memorial service in Jacksonville, on a connecting flight through Charlotte. When I landed in Charlotte, I walked with all my stuff from my JAX arrival gate (D7) to my DFW departure gate (B15). The walk was more stressful than usual because the airport was so crowded.

The moment I set my stuff down at B15, a passenger with expensive clothes and one of those permanent grins established eye contact, pointed his finger at me, and said, “Are you in First?”

Wai... Wha...?

I said, “No, platinum.” My first instinct was to explain that I had a right to occupy the space in which I was standing. It bothers me that this was my first instinct.

He dropped his pointing finger, and his eyes went no longer interested in me. The big grin diminished slightly.

Soon another guy walked up. Same story: the I’m-your-buddy-because-I’m-pointing-my-finger-at-you thing, and then, “First Class?” This time the answer was yes. “ALRIGHT! WHAT ROW ARE YOU IN?” Row two. “AGH,” like he’d been shot in the shoulder. He holstered his pointer finger, the cheery grin became vaguely menacing, and he resumed his stalking.

One guy who got the “First Class?” question just stared back. So, big-grin guy asked him again, “Are you in First Class?” No answer. Big-grin guy leaned in a little bit and looked him square in the eye. Still no answer. So he leaned back out, laughed uncomfortably, and said half under his breath, “Really?...”

I pieced it together watching this big, loud guy explain to his traveling companions so everybody could hear him, he just wanted to sit in Row 1 with his wife, but he had a seat in Row 2. And of course it will be so much easier to take care of it now than to wait and take care of it when everybody gets on the plane.

Of course.

This is the kind of guy who sells things to people. He has probably sold a lot of things to a lot of people. That’s probably why he and his wife have First Class tickets.

I’ll tell you, though, I had to battle against hoping he’d hit his head and fall down on the jet bridge (I battled coz it’s not nice to hope stuff like that). I would never have said something to him; I didn’t want to be Other Jackass to his Jackass. (Although people might have clapped if I had.)

So there’s this surge of emotions, none of them good, going on in my brain over stupid guy in the airport. Sales reps...

This is why Method R Corporation never had sales reps.

But that’s like saying I’ve seen bad aircraft engines before and so now in my airline, I never use aircraft engines. Alrighty then. In that case, I hope you like gliders. And, hey: gliders are fine if that makes you happy. But a glider can’t get me home from Florida. Or even take off by itself.

I wish I sold more Method R software. But never at the expense of being like the guy at the airport. It seems I’d rather perish than be that guy. This raises an interesting question: is my attitude on this topic just a luxury for me that cheats my family and my employees out of the financial rewards they really deserve? Or do I need to become that guy?

I think the answer is not A or B; it’s C.

There are also good sales people, people who sell a lot of things to a lot of people, who are nothing like the guy at the airport. People like Paul Kenny and the honorable, decent, considerate people I work with now at Accenture Enkitec Group who sell through serving others. There were good people selling software at Hotsos, too, but the circumstances of my departure in 2008 prevented me from working with them. (Yes, I do realize: my circumstances would not have prevented me from working with them if I had been more like the guy at the airport.)

This need for duality—needing both the person who makes the creations and the person who connects those creations to people who will pay for them—is probably the most essential of the founder’s dilemmas. These two people usually have to be two different people. And both need to be Good.

In both senses of the word.
06:33 I Can Help You Trace It (13643 Bytes) » Cary Millsap
The first product I ever created after leaving Oracle Corporation in 1999 was a 3-day course about optimizing Oracle performance. The experiences of teaching this course from 2000 through 2003 (heavily revising the material each time I taught it) added up to the knowledge that Jeff Holt and I needed to write Optimizing Oracle Performance (2003).

Between 2000 and 2006, I spent many weeks on the road teaching this 3-day course. I stopped teaching it in 2006. An opportunity to take or teach a course ought to be a joyous experience, and this one had become too much of a grind. I didn’t figure out how to fix it until this year. How I fixed it is the story I’d like to tell you.

The Problem

The problem was simply inefficiency. The inefficiency began with the structure of the course, the 3-day lecture marathon. Realize, 6 × 3 = 18 hours of sitting in a chair, listening attentively to a single voice (my voice) is the equivalent of a 6-week university term of a 3-credit-hour course, taught straight through in three days. No hour-plus homework assignment after each hour of lecture to reinforce the lessons; but a full semester’s worth of listening to one voice, straight through, for three days. What retention rate would you expect from a university course compressed into just 3 days?

So, I optimized. I have created a new course that lasts one day (not even an exhausting full day at that). But how can a student possibly learn as much in 1 day as we used to teach in 3 days? Isn’t a 1-day event bound to be a significantly reduced-value experience?

On the contrary, I believe our students benefit even more now than they used to. Here are the big differences, so you can see why.

The Time Savings

In the 3-day course, I would spend half a day explaining why people should abandon their old system-wide-ratio-based ways of managing system performance. In the new 1-day course, I spend less than an hour explaining the Method R approach to thinking about performance. The point of the new course is not to convince people to abandon anything they’re already doing; it’s to show students the tremendous additional opportunities that are available to them if they’ll just look at what Oracle trace files have to offer. Time savings: 2 hours.

In the 3-day course, I would spend a full day explaining how to interpret trace data. By hand. These were a few little lab exercises, about an hour’s worth. Students would enter dozens of numbers from trace files into laptops or pocket calculators and write results on worksheets. In the new 1-day course, the software tools that a student needs to interpret files of any size—or even directories full of files—are included in the price of the course. Time savings: 5 hours.

In the 3-day course, I would spend half a day explaining how to collect trace data. In the new 1-day course, the software tools that a student needs to get started collecting trace files are included in the price of the course. For software architectures that require more work than our software can do for you, there’s detailed instruction in the course book. Time savings: 3 hours.

In the 3-day course, I would spend half a day working through about five example cases using a software tool to which students would have access for 30 days after they had gone home. In the new 1-day course, I spend one hour working through about eight example cases using software tools that every student will take home and keep forever. I can spend less time per case yet teach more because the cases are thoroughly documented in the course book. So, in class, we focus on the high-level decision making instead of the gnarly technical details you’ll want to look up later anyway. Time savings: 3 hours.

...That’s 13 classroom hours we’ve eliminated from the old 3-day experience. I believe that in these 13 hours, I was teaching material that students weren’t retaining to begin with.

The Book

The next big difference: the book.

In the old 3-day course, I distributed two books: (1) the “Course Notebook,” which was a black and white listing of the course PowerPoint slides, and (2) a copy of Optimizing Oracle Performance (O’Reilly 2003). The O’Reilly book was great, because it contained a lot of detail that you would want to look up after the course. But of course it doesn’t contain any new knowledge we’ve learned since 2003. The Course Notebook, in my opinion, was never worth much to begin with. (In my opinion, no PowerPoint slide printout is worth much to begin with.)

The Mastering Oracle Trace Data (MOTD) book we give each student in my new 1-day course is a full-color, perfect-bound book that explains the course material and far more in deep detail. It is full-color for an important reason. It’s not gratuitous or decorative; it’s because I’ve been studying Edward Tufte. I use color throughout the book to communicate detailed, high-resolution information faster to your brain.

Color in the book helps to reduce student workload and deliver value long after a student has left the classroom. In this class, there is no collection of slide printouts like you’ve archived after every Oracle class you’ve been to since the 1980s. The MOTD book is way better than any other material I’ve ever distributed in my career. I’ve heard students tell their friends that you have to see it to believe it.
“A paper record tells your audience that you are serious, responsible, exact, credible. For deep analysis of evidence and reasoning about complex matters, permanent high-resolution displays [that is, paper] are an excellent start.” —Edward Tufte

The Software

So, where does a student recoup all the time we used to spend going through trace files, and studying how to collect trace data on half a dozen different software architectures? In the thousands of man-hours we’ve invested into the software that we give you when you come to the course. Instead of explaining every little detail about quirks in Oracle trace data that change between Oracle versions 10.1 and 10.2 and 11.2 or 11.2.0.2 and 11.2.0.4, the software does the work for you. Instead of having to explain all the detail work, we have time to explain how to use the results of our software to make decisions about your data.

What’s the catch? Of course, we hope you’ll love our software and want to buy it. The software we give you is completely full-featured and yours to keep forever, but the license limits you to using it only with one login id, and it doesn’t include patches and upgrades, which we release a few times each year. We hope you’ll love our software so much that you’ll want to buy a license that lets you use it on any of your systems and that includes the right to upgrade as we fix bugs and add features. We hope you’ll love it so much that you encourage your colleagues to buy it.

But there’s really no catch. You get software and a course (and a book and a shirt) for less than the daily rate that we used to charge for just a course.

A Shirt?

MOTD London 2011-09-08: “I can help you trace it.”
Yes, a shirt. Each student receives a Method R T-shirt that says, “I can help you trace it.” We don’t give these things away to anyone except for students in my MOTD course. So if you see one, the person wearing it can, in actual fact, Help You Trace It.

The Net Result

The net result of all this optimization is benefits on several fronts:
  • The course costs a lot less than it used to. The fee is presently only about 25% of the 3-day course’s price, and the whole experience requires less than ⅓ of time away from work that the original course did.
  • In the new course, our students don’t have to work so hard to make productive use of the course material. The book and the software take so much of the pressure off. We do talk about what the fields in raw trace data mean—I think it’s necessary to know that in order to use the data properly, and have productive debates with your sys/SAN/net/etc. administration colleagues. But we don’t spend your time doing exercises to untangle nested (recursive) calls by hand. The software you take home does that for you. That’s why it is so much easier for a student to put this course to work right away.
  • Since the course duration is only one day, I can visit far more cities and meet far more students each year. That’s good for students who want to participate, and it’s great for me, because I get to meet more people.

Plans

The only thing missing from our Mastering Oracle Trace Data course right now is you. I have taught the event now in Southlake, Texas (our home town), in Copenhagen, and in London. It’s field-tested and ready to roll. We have several cities on my schedule right now. I’ll be teaching the course in Birmingham UK on the day after UKOUG wraps up, December 8. I’ll be doing Orlando and Tampa in mid-December. I’ll teach two courses this coming January in Manhattan and Long Island. There’s Billund (Legoland) DK in April. We have more plans in the works for Seattle, Portland, Dallas, and Cleveland, and we’re looking for more opportunities.

Share the word by linking the official
MOTD sticker to http://method-r.com/.
My wish is for you to help me book more cities in North America and Europe (I hope to expand beyond that soon). If you are part of a company or a user group with colleagues who would be interested in attending the course, I would love to hear from you. Registering en masse saves you money. The magic number for discounting is 10 students on a single registration from one company or user group.

I can help you trace it.

06:33 Gwen Shapira on SSD (997 Bytes) » Cary Millsap
If you haven’t seen Gwen Shapira’s article about de-confusing SSD, I recommend that you read it soon.

One statement stood out as an idea on which I wanted to comment:
If you don’t see significant number of physical reads and sequential read wait events in your AWR report, you won’t notice much performance improvements from using SSD.
I wanted to remind you that you can do better. If you do notice a significant number of physical reads and sequential write wait events in your AWR report, then it’s still not certain that SSD will improve the performance of the task whose performance you’re hoping to improve. You don’t have to guess about the effect that SSD will have upon any business task you care about. In 2009, I wrote a blog post that explains.
06:33 Fail Fast (5502 Bytes) » Cary Millsap
Among movements like Agile, Lean Startup, and Design Thinking these days, you hear the term fail fast. The principle of failing fast is vital to efficiency, but I’ve seen project managers and business partners be offended or even agitated by the term fail fast. I’ve seen it come out like, “Why the hell would I want to fail fast?! I don’t want to fail at all.” The implication, of course: “Failing is for losers. If you’re planning to fail, then I don’t want you on my team.”

I think I can help explain why the principle of “fail fast” is so important, and maybe I can help you explain it, too.

Software developers know about fail fast already, whether they realize it or not. Yesterday was a prime example for me. It was a really long day. I didn’t leave my office until after 9pm, and then I turned my laptop back on as soon as I got home to work another three hours. I had been fighting a bug all afternoon. It was a program that ran about 90 seconds normally, but when I tried a code path that should have been much faster, I could let it run 50 times that long and it still wouldn’t finish.

At home, I ran it again and left it running while I watched the Thunder beat the Spurs, assuming the program would finish eventually, so I could see the log file (which we’re not flushing often enough, which is another problem). My MacBook Pro ran so hard that the fan compelled my son to ask me why my laptop was suddenly so loud. I was wishing the whole time, “I wish this thing would fail faster.” And there it is.

When you know your code is destined to fail, you want it to fail faster. Debugging is hard enough as it is, without your stupid code forcing you to wait an hour just to see your log file, so you might gain an idea of what you need to go fix. If I could fail faster, I could fix my problem earlier, get more work done, and ship my improvements sooner.

But how does that relate to wanting my business idea to fail faster? Well, imagine that a given business idea is in fact destined to fail. When would you rather find out? (a) In a week, before you invest millions of dollars and thousands of hours investing into the idea? Or (b) In a year, after you’ve invested millions of dollars and thousands of hours?

I’ll take option (a) a million times out of a million. It’s like asking if I’d like a crystal ball. Um, yes.

The operative principle here is “destined to fail.” When I’m fixing a reported bug, I know that once I create reproducible test case for that bug, my software will fail. It is destined to fail on that test case. So, of course, I want for my process of creating the reproducible test case, my software build process, and my program execution itself to all happen as fast as possible. Even better, I wish I had come up with the reproducible test case a year or two ago, so I wouldn’t be under so much pressure now. Because seeing the failure earlier—failing fast—will help me improve my product earlier.

But back to that business idea... Why would you want a business idea to fail fast? Why would you want it to fail at all? Well, of course, you don’t want it to fail, but it doesn’t matter what you want. What if it is destined to fail? It’s really important for you to know that. So how can you know?

Here’s a little trick I can teach you. Your business idea is destined to fail. It is. No matter how awesome your idea is, if you implement your current vision of some non-trivial business idea that will take you, say, a month or more to implement, not refining or evolving your original idea at all, your idea will fail. It will. Seriously. If your brain won’t permit you to conceive of this as a possibility, then your brain is actually increasing the probability that your idea will fail.

You need to figure out what will make your idea fail. If you can’t find it, then find smart people who can. Then, don’t fear it. Don’t try to pretend that it’s not there. Don’t work for a year on the easy parts of your idea, delaying the inevitable hard stuff, hoping and praying that the hard stuff will work its way out. Attack that hard stuff first. That takes courage, but you need to do it.

Find your worst bottleneck, and make it your highest priority. If you cannot solve your idea’s worst problem, then get a new idea. You’ll do yourself a favor by killing a bad idea before it kills you. If you solve your worst problem, then find the next one. Iterate. Shorter iterations are better. You’re done when you’ve proven that your idea actually works. In reality. And then, because life keeps moving, you have to keep iterating.

That’s what fail fast means. It’s about shortening your feedback loop. It’s about learning the most you can about the most important things you need to know, as soon as possible.

So, when I wish you fail fast, it’s a blessing; not a curse.
06:33 And now, ...the video (1951 Bytes) » Cary Millsap
First, I want to thank everyone who responded to my prior blog post and its accompanying survey, where I asked when video is better than a paper. As I mentioned already in the comment section for that blog post, the results were loud and clear: 53.9% of respondents indicated that they’d prefer reading a paper, and 46.1% indicated that they’d prefer watching a video. Basically a clean 50/50 split.

The comments suggested that people have a lower threshold for “polish” with a video than with a paper, so one of the ideas to which I’ve needed to modify my thinking is to just create decent videos and publish them without expending a lot of effort in editing.

But how?

Well, the idea came to me in the process of agreeing to perform a 1-hour session for my good friends and customers at The Pythian Group. Their education department asked me if I’d mind if they recorded my session, and I told them I would love if they would. They encouraged me to open it up the the public, which of course I thought (cue light bulb over head) was the best idea ever. So, really, it’s not a video as much as it’s a recorded performance.

I haven’t edited the session at all, so it’ll have rough spots and goofs and imperfect audio... But I’ve been eager ever since the day of the event (2013-02-15) to post it for you.

So here you go. For the “other 50%” who prefer watching a video, I present to you: “The Method R Profiling Ecosystem.” If you would like to follow along in the accompanying paper, you can get it here: “A First Look at Using Method R Workbench Software.”

06:33 An Organizational Constraint that Diminishes Software Quality (4847 Bytes) » Cary Millsap
One of the biggest problems in software performance today occurs when the people who write software are different from the people who are required to solve the performance problems that their software causes. It works like this:
  1. Architects design a system and pass the specification off to the developers.
  2. The developers implement the specs the architects gave them, while the architects move on to design another system.
  3. When the developers are “done” with their phase, they pass the code off to the production operations team. The operators run the system the developers gave them, while the developers move on to write another system.
The process is an assembly line for software: architects specialize in architecture, developers specialize in development, and operators specialize in operating. It sounds like the principle of industrial efficiency taken to its logical conclusion in the software world.


In this waterfall project plan,
architects design systems they never see written,
and developers write systems they never see run.

Sound good? It sounds like how Henry Ford made a lot of money building cars... Isn’t that how they build roads and bridges? So why not?

With software, there’s a horrible problem with this approach. If you’ve ever had to manage a system that was built like this, you know exactly what it is.

The problem is the absence of a feedback loop between actually using the software and building it. It’s a feedback loop that people who design and build software need for their own professional development. Developers who never see their software run don’t learn enough about how to make their software run better. Likewise, architects who never see their systems run have the same problem, only it’s worse, because (1) their involvement is even more abstract, and (2) their feedback loops are even longer.

Who are the performance experts in most Oracle shops these days? Unfortunately, it’s most often the database administrators, not the database developers. It’s the people who operate a system who learn the most about the system’s design and implementation mistakes. That’s unfortunate, because the people who design and write a system have so much more influence over how a system performs than do the people who just operate it.

If you’re an architect or a developer who has never had to support your own software in production, then you’re probably making some of the same mistakes now that you were making five years ago, without even realizing they’re mistakes. On the other hand, if you’re a developer who has to maintain your own software while it’s being operated in production, you’re probably thinking about new ways to make your next software system easier to support.

So, why is software any different than automotive assembly, or roads and bridges? It’s because software design is a process of invention. Almost every time. When is the last time you ever built exactly the same software you built before? No matter how many libraries you’re able to reuse from previous projects, every system you design is different from any system you’ve ever built before. You don’t just stamp out the same stuff over and over.

Software is funny that way, because the cost of copying and distributing it is vanishingly small. When you make great software that everyone in the world needs, you write it once and ship it at practically zero cost to everyone who needs it. Cars and bridges don’t work that way. Mass production and distribution of cars and bridges requires significantly more resources. The thousands of people involved in copying and distributing cars and bridges don’t have to know how to invent or refine cars or bridges to do great work. But with software, since copying and distributing it is so cheap, almost all that’s left is the invention process. And that requires feedback, just like inventing cars and bridges did.

Don’t organize your software project teams so that they’re denied access to this vital feedback loop.
06:32 work with a custom schema with Spark loading CSV file (33571 Bytes) » Developer 木匠
Here's how you can work with a custom schema, a complete demo:
Unix shell code,
$>
echo "
Slingo, iOS
Slingo, Android
" > game.csv
Scala code:
import org.apache.spark.sql.types._

val customSchema = StructType(Array(
StructField("game_id", StringType, true),
StructField("os_id", StringType, true)
))

val csv_df = spark.read.format("csv").schema(customSchema).load("game.csv")
csv_df
.show

csv_df
.orderBy(asc("game_id"), desc("os_id")).show
csv_df
.createOrReplaceTempView("game_view")
val sort_df = sql("select * from game_view order by game_id, os_id desc")
sort_df
.show

Reference,
06:32 spark 2.2.1 installation guide on Mac (1606 Bytes) » Developer 木匠
// blog install spark 2.2.1 on Mac

Spark is incompatible with Java 9.
Install Java 8 instead.

# to uninstall Java 9
brew cask uninstall java

Here is the complete and simple installation steps for spark on Mac :

brew tap caskroom/versions
brew cask search java
brew cask install java8

# brew install scala # You might need these step,
brew install apache-spark


# Start spark shell,

./bin/spark-shell --master local[2]

./bin/spark-shell --master local[1]
06:32 flex CSV loader (6213 Bytes) » Developer 木匠
Purpose: Load any structure CSV file in Meteor to MongoDB, through package harrison:papa-parse.

meteor add harrison:papa-parse
meteor add themeteorchef:bert
meteor add reactive-var
meteor add fortawesome:fontawesome

$> meteor create csv

Then copy below 2 files into ./csv/

  • csv.html

<head>
  <title>cvs</title>
</head>

<body>
  <h1>Welcome to CSV loader!</h1>

  {{> upload}}
</body>

<template name="upload">
  <h4 class="page-header">Upload a CSV, unfixed structure.</h4>

  {{#unless uploading}}
    <input type="file" name="upload_CSV">
  {{else}}
    <p><i class="fa fa-spin fa-refresh"></i> Uploading files...</p>
  {{/unless}}
</template>


  • csv.js

Order = new Mongo.Collection('order');
if (Meteor.isClient) {
  Template.upload.onCreated( () => {
    Template.instance().uploading = new ReactiveVar( false );
  });

  Template.upload.helpers({
    uploading() {
      return Template.instance().uploading.get();
    }
  });

  Template.upload.events({
    'change [name="upload_CSV"]' (event, template ) {
      Papa.parse(event.target.files[0], {
        header: true,
        complete(results, file) {
          Meteor.call('parseUpload', results.data, (error, response) => {
            if (error) {
              Bert.alert(error.reason, 'warning');
            } else {
              template.uploading.set(false);
              Bert.alert('Upload complete!', 'success', 'growl-top-right' );
            }
          });
        }
      });
    }
  });

}
if (Meteor.isServer) {
  Meteor.startup(function () {
    // code to run on server at startup
  });

  Meteor.methods({
    parseUpload(data){
      //check(data, Array);

      for ( let i = 0; i < data.length; i++) {
        let item = data[i],
            exists = Order.findOne({orderId: item.order_no});
        if ( (item.order_no === undefined) || !exists ) {
          Order.insert(item);
        } else {
          console.warn( 'Rejected. This item already exists.');
          console.log(exists.Age + "," + item.order_no);
        }
      }
    }
  });
}
06:32 coding interview chat 面试算法座谈 (3169 Bytes) » Developer 木匠
各位老大好,

近期咱不定期的周一周三,进行面试算法解题讲座, 针对一流软件公司的面试题。
欢迎前来当听众和提问。

今天会谈到 基本数据结构:binary heap.
中级数据结构: hash heap, segment tree
和高级数据结构 segment tree with lazy propagation
(各种数据结构都是一层层深入演进,咱尽力解释的简单清晰).

会谈到一个算法: 扫描线算法,

有了这些基础知识以后,会打代码解答一个超高难度问题: 

heap 推荐学习资料:

06:32 code review (8027 Bytes) » Developer 木匠
The biggest rule is that the point of code review is to find problems in code before it gets committed - what you're looking for is correctness.
The most common mistake in code review - the mistake that everyone makes when they're new to it - is judging code by whether it's what the reviewer would have written.
Remember that there are many ways to solve a problem.

Advice


  • What you're looking for is correctness.

Programming decisions are a matter of opinion. Reviewers and developers should seek to understand each other’s perspective but shouldn’t get into a philosophical debate.
  • Be humble.

Don’t be a prick.  :-)
“I didn’t see where these variables were initialized”.
“What do you think about Standard DRY and if it’s applies here?”, 
“I don’t understand why this is a global variable “
  • Make sure you have coding standards in place.

Coding standards are shared set of guidelines in an organization with buy-in from everyone. If you don’t have coding standards, then don’t let the discussion turn into a pissing contest over coding styles (opening braces ‘{‘ on the same line or the next!) If you run into a situation like that, take the discussion offline to your coding standards forum.
One of the goals of code reviews is to create ‘maintainable’ software.
  • Learn to communicate well.

You must be able to clearly express your ideas and reasons.
  • Authors should annotate source code before the review.

Authors should annotate code, what is the problem and goaldesign and solution implementation, before the review occurs, because annotations guide the reviewer through the changes, showing which files to look at first and defending the reason behind each code modification. Annotations should be directed at other reviewers to ease the process and provide more depth in context. As an added benefit, the author will often find additional errors before the peer review even begins. More bugs found prior to peer review will yield in lower defect density because fewer bugs exist overall.
.
.
Effective code reviews require a healthy culture that values quality and excellence. Code reviews will not give you the desired results if the team doesn’t believe in delivering high-quality products. You need a positive culture in which people are engaged – one that thrives on constructive criticism and allows the best ideas to win.
Peer code review can be efficient and effective while fostering open communication and knowledge-share between teammates.

Reference:


06:32 When ANSI join comes to play? (3452 Bytes) » Developer 木匠
ANSI join has been around for many years.
There are a couple of situations better off using ANSI join rather than Oracle native join.

1. Join many tables.

Reason:

Use explicit JOINs rather than implicit (regardless whether they are outer joins or not) is that it's much easier to accidentally create a cartesian product with the implicit joins. With explicit JOINs you cannot "by accident" create one. The more tables are involved the higher the risk is that you miss one join condition.

2. Outer join.

Reason:

Basically (+) is severely limited compared to ANSI joins. ANSI syntax is cleaner - you are not going to normal join if you forget (+) in some multi-column outer join.
In the past there were some bugs with ANSI syntax but if you go with latest 11.2 or 12.1 that should be fixed already.


Example:

SELECT *
  FROM table_a a 
  JOIN table_b b 
    ON a.col1 = b.col1
  JOIN table_c c
    ON b.col2 = c.col2;

Example 2:

SELECT d.department_id, e.last_name, j.job_name
  FROM departments d 
  LEFT OUTER JOIN employees e
    ON d.department_id = e.department_id
  LEFT OUTER JOIN job j
    ON e.job_id = j.job_id
;


06:32 Thoughts and an example on Expression-Oriented Programming (4297 Bytes) » Developer 木匠
Here is the code, to read password from .pgpass to connect o a PostgreSQL database.

object DbUtil {
  def dbPassword(hostname:String, port:String, database:String, username:String ):String = {
    // Usage: val thatPassWord = dbPassword(hostname,port,database,username)
    // .pgpass file format, hostname:port:database:username:password
    val passwdFile = new java.io.File(scala.sys.env("HOME"), ".pgpass")
    var passwd = ""
    val fileSrc = scala.io.Source.fromFile(passwdFile)
    fileSrc.getLines.foreach{line =>
      val connCfg = line.split(":")
      if (hostname == connCfg(0)
        && port == connCfg(1)
        && database == connCfg(2)
        && username == connCfg(3)
      ) { 
        passwd = connCfg(4)
      }
    }
    fileSrc.close
    passwd
  }

  def passwordFromConn(connStr:String) = {
    // Usage: passwordFromConn("hostname:port:database:username")
    val connCfg = connStr.split(":")
    dbPassword(connCfg(0),connCfg(1),connCfg(2),connCfg(3))
  }
}

Thoughts.

* foreach v.s.  map().filter()(0) . e.g.: List(1,2,3).filter(_ > 0)(0)
* var passwd, v.s.  val passwd inside of foreach.
* is it good to do chain of method style?

* how to break when find one, in a for loop ?

import util.control.Breaks._
breakable {
for (i <- 1 to 10) {
println(i)
if (i > 4) break // break out of the for loop }
}

* other improvement ?

06:32 Scala access PostgresQL database complete demo (9298 Bytes) » Developer 木匠
// Connect o PostgresQL

$>
brew info postgres

// Start manually:

pg_ctl -D /usr/local/var/postgres start

// or, write stats to log,
pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start

// Stop manually:

pg_ctl -D /usr/local/var/postgres stop

// Status

pg_ctl -D /usr/local/var/postgres status


createdb scala_db
createuser scala_user
psql -U scala_user -d scala_db

SQL>

create table emps
(
id integer,
last_name varchar(100),
email varchar(200),
gender varchar(10),
department varchar(100),
start_date date,
salary integer,
job_title varchar(100),
region_id int,
primary key (id)
);

insert into emps values (1,'Kelley','rkelley0@sours.getString("country")ndcloud.com','Female','Computers','10/2/2009',67470,'Structural Engineer',2);
insert into emps values (2,'Armstrong','sarmstrong1@infoseek.co.jp','Male','Home','3/31/2008',71869,'Financial Advisor',2);
insert into emps values (3,'Carr','fcarr2@woothemes.com','Male','Home','7/12/2009',101768,'Recruiting Manager',3);
insert into emps values (4,'Murray','jmurray3@gov.uk','Female','Jewelery','12/25/2014',96897,'Desktop Support Technician',3);
insert into emps values (5,'Ellis','jellis4@sciencedirect.com','Female','Home','9/19/2002',63702,'Software Engineer III',7);


$>

// scala -classpath ~/app/lib/postgresql-42.1.1.jre6.jar
bin/spark-shell -classpath ~/app/lib/postgresql-42.1.1.jre6.jar

// Scala code,
import java.sql.Driver
import java.sql.DriverManager
import java.sql.Connection

val driver = "org.postgresql.Driver"
val url = "jdbc:postgresql://localhost/scala_db?user=scala_user"
Class.forName(driver)
var connection: Connection = null

connection = DriverManager.getConnection(url)
val statement = connection.createStatement()
val resultSet = statement.executeQuery("select * from emps limit 5")
resultSet.next
resultSet.getString("last_name")
resultSet.getString("department")

while ( resultSet.next() ) {
val last_name = resultSet.getString("last_name")
val dept = resultSet.getString("department")
printf("%s: %s \n", dept, last_name)
}

// Prepare SQL statement
val queryStr = "select * from emps where department = ? "
val ps = connection.prepareStatement(queryStr)
ps.setString(1, "Home")
val rs = ps.executeQuery()

while ( rs.next() ) {
val last_name = rs.getString("last_name")
val dept = rs.getString("department")
val startDate = rs.getString("start_date")
printf("%s: %s \t %s \n", dept, last_name, startDate)
}

06:32 Running jupyter notebook with pyspark shell on Windows 10 (1745 Bytes) » Developer 木匠
# Running jupyter notebook with pyspark shell.

This notebook will not run in an ordinary jupyter notebook server.
Here is how we can load pyspark to use Jupyter notebooks.
On Mac, Linux, something like this will work:
On Windows 10, it works at 2017-Nov-21.

# Install Anaconda

https://www.anaconda.com/download/

## Run "Anaconda Prompt" to open command line:

%windir%\System32\cmd.exe "/K" E:\Users\datab_000\Anaconda3\Scripts\activate.bat E:\Users\datab_000\Anaconda3

Note: above commands are in one single line.

## Then set system environment variables:

rem set PYSPARK_PYTHON=python3
set PYSPARK_DRIVER_PYTHON=jupyter
set PYSPARK_DRIVER_PYTHON_OPTS=notebook

## start pyspark

E:\movie\spark\spark-2.2.0-bin-hadoop2.7\bin\pyspark.cmd


Reference:

06:32 Refactoring PL/SQL Table Function demo (4529 Bytes) » Developer 木匠
"Refactoring is the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure." -- Martin Fowler in Refactoring: Improving The Design Of Existing Code

Here is an example how I build PL/SQL table function usually.

Before refactoring the code, we have to write the SQL in 2 places, it is a repetition.

create or replace PACKAGE demo_pkg
AS
  CURSOR gc_data IS select 1 col1, 2 col2 from dual;
  type data_tab is table of gc_data%RowType;

  FUNCTION f1(p_a number)
    RETURN data_tab PIPELINED;
end;
/


CREATE OR REPLACE PACKAGE BODY demo_pkg
AS
  FUNCTION f1(p_a number)
    RETURN data_tab PIPELINED
  is
  BEGIN
    FOR lr in (select 1 col1, 2 col2 from dual where p_a > 5)
    LOOP
      pipe ROW(lr);
    END LOOP;
    RETURN;
  EXCEPTION
  WHEN OTHERS THEN
    /* Don't forget to clean up here, too! */
    /* Standard error logging mechanism */
    --logger.log_error('Unhandled Exception', l_scope);
    raise;
  END;
end;
/


After refactoring the code, we only need to write the SQL(cursor) one time,  removed unnecessary repetition.

create or replace PACKAGE demo_pkg
AS
  CURSOR gc_data (p_a number) IS

    select 1 col1, 2 col2, 3 col3 from dual where p_a > 5;
  type data_tab is table of gc_data%RowType;

  FUNCTION f1(p_a number)
    RETURN data_tab PIPELINED;
end;
/


CREATE OR REPLACE PACKAGE BODY demo_pkg
AS
  FUNCTION f1(p_a number)
    RETURN data_tab PIPELINED
  is
    lr gc_data%rowtype;
  BEGIN
    OPEN gc_data(p_a);
    LOOP
      FETCH gc_data INTO lr;
      pipe ROW(lr);
      EXIT WHEN gc_data%NotFound;
    END LOOP;
    close gc_data;
    RETURN;
  EXCEPTION
  WHEN OTHERS THEN
    /* Don't forget to clean up here, too! */
    /* Standard error logging mechanism */
    --logger.log_error('Unhandled Exception', l_scope);
    raise;
  END;
end;
/


Observed:

This makes the output DRY, but for input, if we add/update/remove parameters, we have to update 2 places.

Reference:

1) http://www.refactoring.com/
2) call example: select col1, col2 from table(demo_pkg.f1(18));
06:32 Python read config .ini and connect to a database (383 Bytes) » Developer 木匠
This is my first Python application.

There are some knowledge points inline with code.

..
The souce code is here, https://github.com/goodgoodwish/code_tip/blob/master/python/db.py.
.
.
06:32 Load CSV file in Meteor (4724 Bytes) » Developer 木匠
Purpose: Demo how to load a CVS file into MongoDB.

$> meteor create load_csv

Then copy below 2 files into ./load_csv/

load_csv.html

<head>
  <title>load_csv</title>
</head>

<body>
 <h1>Welcome to CSV loader!</h1>
 {{> read_file_orders}}
 {{> order}}
</body>

<template name="read_file_orders">
 <form class="well form-inline">
  <input class="input-file" id="fileInput2" type="file" name="files[]">
  <Button class="btn btn-primary" id="read_orders">Import</button>
  <button>clear file name</button>
 </form>
</template>

<template name="order">
  <ul>
  {{#each order}}
    <li>{{id}}: {{value}}</li>
  {{/each}}
  </ul>
</template>


load_csv.js

Orders = new Mongo.Collection("orders");
if (Meteor.isClient) {
// counter starts at 0

  Template.read_file_orders.events({
   "click #read_orders" : function(e) {
    var f = document.getElementById('fileInput2').files[0];
    console.log("read file");
    readFile(f, function(content) {
      Meteor.call('upload',content);
    });
   }
  });

  Template.order.helpers({
    'order': function(){
     return Orders.find();
    }
  });

  readFile = function(f,onLoadCallback) {
    //When the file is loaded the callback is called with the contents as a string
    var reader = new FileReader();
    reader.onload = function (e){
      var contents=e.target.result
      onLoadCallback(contents);
    }
    reader.readAsText(f);
  };
}

if (Meteor.isServer) {
  Meteor.startup(function () {
  // code to run on server at startup

  });
  Meteor.methods({
    upload : function(fileContent) {
     console.log("start insert");
     import_file_orders(fileContent);
     console.log("completed");
    }
  });

  import_file_orders = function(file) {
   console.log("enter function import_file_orders")
   var lines = file.split(/\r\n|\n/);
   var l = lines.length;
   for (var i=0; i < l; i++) {
    var line = lines[i];
    var line_parts = line.split(',');
    var id = line_parts[0];
    var value = line_parts[1];
    var result = Orders.insert({id:id, value:value});
    console.log(Orders.findOne(result));
   }
  };

}
06:32 Join 3 tables - Oracle native outer join v.s. ANSI outer join (6109 Bytes) » Developer 木匠
select t1.id t1_id, t2.id t2_id, t3.id t3_id
from t1,t2,t3
where t1.id = t2.id(+)
and t2.id = t3.id(+)
--order by 1
;

     T1_ID      T2_ID      T3_ID
---------- ---------- ----------
         1          1          1
         3                      
         2          2           


select t1.id t1_id, t2.id t2_id, t3.id t3_id
from t1
left join t2 on t1.id = t2.id
left join t3 on t3.id = t2.id
--order by 1
;

     T1_ID      T2_ID      T3_ID
---------- ---------- ----------
         1          1          1
         3                      
         2          2           

Observation: They return same result.


Fyi,  init testing data.

drop table t1 purge;
drop table t2 purge;
drop table t3 purge;
create table t1 (id number);
create table t2 (id number);
create table t3 (id number);
insert into t1(id) select rownum from dual connect by level <= 3;
insert into t2(id) select rownum from dual connect by level <= 2;
insert into t3(id) select rownum from dual connect by level <= 1;
commit;

06:32 JavaScript Closures: setTimeout and loop (6692 Bytes) » Developer 木匠
Problem:

for (var i = 0; i < 3; i++) {
    setTimeout(function() {
        console.log("My value: " + i);
    }, 1);
}

My value: 3
My value: 3
My value: 3

}

Why:

Variable Global scope, and JavaScript asynchronous event loop single thread execution.

Solution:

// Block scope, ES2005
for (let i = 0; i < 3; i++) {
    setTimeout(function() {
        console.log("My value: " + i);
    }, 1000);
}

//Immediately Invoked creation, and function scope.
for (var i = 0; i < 3; i++) {
    setTimeout((function(pNum) {
        return function() {
            console.log('i: ', pNum)
        }
    })(i), 1000);
}

//  IIFE (Immediately Invoked Function Expression)
for (var i = 0; i < 3; i++) {
    (function(pNum = i) {
        setTimeout(
            function() {
                console.log('i : ', pNum)
            }, 1000)
    })(i);
}


Enjoy the sunshine,


06:32 INSERT ALL and sequence NextVal (2812 Bytes) » Developer 木匠
drop table t1 purge;
drop table t2 purge;
drop table t3 purge;

create table t1(n1 number, n2 number);
create table t2(n1 number, n2 number);
create table t3(n1 number, n2 number);

create sequence s1;

Problem:

INSERT ALL
    INTO t1(n1, n2) values(n1_seq, nn)
    INTO t2(n1, n2) values(n1_seq, nn) 
select s1.nextval n1_seq, rownum*10 nn from dual connect by level <= 2;

Error report -
SQL Error: ORA-02287: sequence number not allowed here
02287. 00000 -  "sequence number not allowed here"
*Cause:    The specified sequence number (CURRVAL or NEXTVAL) is inappropriate
           here in the statement.
*Action:   Remove the sequence number.

Fix:

INSERT ALL
    INTO t1(n1, n2) values(s1.nextval, nn)
    INTO t2(n1, n2) values(s1.currval, nn) 
    INTO t3(n1, n2) values(s1.currval, nn) 
select rownum*10 nn from dual connect by level <= 2;

select * from t1;
select * from t2;
select * from t3;

06:32 Golang range and close channel (1829 Bytes) » Developer 木匠
Code:

package main

import "golang.org/x/tour/tree"
import "fmt"

// Walk walks the tree t sending all values
// from the tree to the channel ch.

func Walk(t *tree.Tree, ch chan int) {
 if t == nil {
  fmt.Println("Null")
  return
 }
 if t.Left != nil {
  Walk(t.Left, ch)
 }
 ch <- t.Value
 fmt.Println("send ", t.Value)
 if t.Right != nil {
  Walk(t.Right, ch)
 }
}



func main() {
 ta := tree.New(1)
 ca := make(chan int)
 go func() {
  Walk(ta, ca)
  close(ca)
 }()


 for v := range ca {
  fmt.Println("get ", v)
 }
}


Comments:

See which line the channel is closed.
06:32 Golang connect to Oracle database (8189 Bytes) » Developer 木匠
Purpose: setup Go Oracle driver, and a complete code to demo some basic DDL, DML, and bulk batching processing.

Minimum requirements are
- Go 1.3 with CGO enabled,
- a GCC C compiler,
- and Oracle 11g (11.2.0.4.0) or Oracle Instant Client (11.2.0.4.0).

This demo is tested on Go 1.5.2, Oracle Virtual Box, imported OTN_Developer_Day_VM.ova appliance/image file.

** environment settings **

------- Linux ----------
export GOROOT=/usr/local/go
#export GOPATH=/home/oracle/go
export GOPATH=/media/sf_GTD/Project/Go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
mkdir -p $GOPATH/src/github.com/user/hello
cd $GOPATH/src/github.com/user/hello
echo $PATH
echo $GOPATH
# Oracle client driver, environment parameters
# Set the CGO_CFLAGS and CGO_LDFLAGS environment variables to locate the OCI headers and library,
export CGO_CFLAGS=-I$ORACLE_HOME/rdbms/public
export CGO_LDFLAGS="-L$ORACLE_HOME/lib -lclntsh"

-----------
-- install Oracle driver --
go get github.com/rana/ora

-- manual install driver from source zip files --
download source zip files, and extract into $GOPATH/src/gopkg.in/rana/ora.v3/ , or rename ora-master to ora.v3

download address: https://github.com/rana/ora/tree/v3

The files structure will be looks like,
src\gopkg.in\rana\ora.v3\
                         ora.go
                         env.go
                         ...
                         \examples\...


To import this package, add the following line to your code:
import "gopkg.in/rana/ora.v3"

Code:

// Copyright 2015, Author: Charlie Database Craftsman. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

/*
Package main demo OCI array interface.
It connects to a database, create table, insert data, and query the data.

E:\GTD\Project\Go\src\github.com\user\hello\oracle_demo.go
*/
package main


import (
 "fmt"
 "gopkg.in/rana/ora.v3"
)


func main() {
 // example usage of the ora package driver
 // connect to a server and open a session
 env, err := ora.OpenEnv(nil)
 defer env.Close()
 if err != nil {
  panic(err)
 }
 srvCfg := ora.NewSrvCfg()
 srvCfg.Dblink = "//localhost/orcl"
 srv, err := env.OpenSrv(srvCfg)
 defer srv.Close()
 if err != nil {
  panic(err)
 }
 sesCfg := ora.NewSesCfg()
 sesCfg.Username = "scott"
 sesCfg.Password = "tiger"
 ses, err := srv.OpenSes(sesCfg)
 fmt.Println("connected")
 defer ses.Close()
 if err != nil {
  panic(err)
 }

 //StmtCfg.PrefetchRowCount = 1000
 stmtTbl, err := ses.Prep(
  `declare
  l_sql varchar2(32767);
  l_cnt pls_integer;
begin
  l_sql := 'drop TABLE emp_go purge';
  select count(*) into l_cnt from user_tables where table_name='EMP_GO';
  if l_cnt > 0 then
    execute immediate l_sql;
  end if;
end;`)
 defer stmtTbl.Close()
 if err != nil {
  panic(err)
 }
 rowsAffected, err := stmtTbl.Exe()
 if err != nil {
  panic(err)
 }
 fmt.Println(rowsAffected, " rows Affected. drop table emp_go if exists.")

 rowsAffected, err = ses.PrepAndExe("CREATE TABLE emp_go (empno number(5,0), ename VARCHAR2(50))")
 if err != nil {
  panic(err)
 }
 fmt.Println("table emp_go created")

 tx1, err := ses.StartTx()
 rowsAffected, err = ses.PrepAndExe("delete emp_go")
 tx1.Commit()

 stmt, err := ses.Prep("INSERT INTO emp_go (empno, ename) VALUES (:N1, :C1)")
 defer stmt.Close()
 rowsAffected, err = stmt.Exe(uint64(1001), "Charlie")
 rowsAffected, err = stmt.Exe(uint64(1002), "Vicky")
 if err != nil {
  panic(err)
 }
 fmt.Println(rowsAffected, "add")

 tx1.Commit()
 tx1.Rollback()
 fmt.Println("commit")

 fmt.Println("Demo fetch records")
 stmtQry, err := ses.Prep("SELECT empno, ename FROM emp_go")
 defer stmtQry.Close()
 if err != nil {
  panic(err)
 }
 rset, err := stmtQry.Qry()
 if err != nil {
  panic(err)
 }
 for rset.Next() {
  fmt.Println(rset.Row[0], rset.Row[1])
 }
 if rset.Err != nil {
  panic(rset.Err)
 }
}


Output:

$> go run oracle_demo.go

connected
0  rows Affected. drop table emp_go if exists.
table emp_go created
1 add
commit
Demo fetch records
1001 Charlie
1002 Vicky

.

Reference:
https://github.com/rana/ora/blob/master/README.md
http://gopkg.in/rana/ora.v3

06:32 Golang Oracle Performance Extensions (17890 Bytes) » Developer 木匠
Purpose: Demo Go Oracle database driver OCI Performance Extensions.

We will do a benchmark and show you the improvement before and after enabling Prefetching rows.
Also use SQL session trace file to prove that Update Batching works, than is insert many rows at one time.

Info:

Update Batching

You can reduce the number of round-trips to the database, thereby improving application performance, by grouping multiple UPDATE, DELETE, or INSERT statements into a single batch and having the whole batch sent to the database and processed in one trip. This is referred to as 'update batching'.

Prefetching rows

This reduces round-trips to the database by fetching multiple rows of data each time data is fetched. The extra data is stored in client-side buffers for later access by the client. The number of rows to prefetch can be set as desired.
...

Code:

// Copyright 2015, Author: Charlie Database Craftsman. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

/*
Package main demo 2 Oracle OCI Performance Extensions: array interface and Prefetching rows.

- Update Batching
It calls StmtCfg.SetPrefetchRowCount to set Prefetching size for SELECT query.
- Prefetching rows
It is binding slice type data, to pass array to database in single SQL to do Bulk DML, in one context switch.

Notes:
stmtQry.Cfg().SetPrefetchRowCount(uint32(200))  works,
ses.Cfg().StmtCfg.SetPrefetchRowCount(2000) works,

EnvCfg -> SrvCfg are both not work, because code issue, NewStmtCfg() overwrite Env/Srv on session level.
Env.StmtCfg.SetPrefetchRowCount not working.


| Modification History:
|  Date        Who          What
| 11-Jan-2016: Charlie(Yi): Abstract checkErr function,
| 18-Dec-2015: Charlie(Yi): create the package,

*/
package main


import (
 "fmt"
 "gopkg.in/rana/ora.v3"
)


func main() {
 // example usage of the ora package driver
 // connect to a server and open a session
 env, err := ora.OpenEnv(nil)
 defer env.Close()
 checkErr(err)

 srvCfg := ora.NewSrvCfg()
 srvCfg.Dblink = "//localhost/orcl"

 srv, err := env.OpenSrv(srvCfg)
 defer srv.Close()
 checkErr(err)

 sesCfg := ora.NewSesCfg()
 sesCfg.Username = "scott"
 sesCfg.Password = "tiger"
 fmt.Println("Session PrefetchRowCount :", sesCfg.StmtCfg.PrefetchRowCount())
 sesCfg.StmtCfg.SetPrefetchRowCount(uint32(2000))
 ses, err := srv.OpenSes(sesCfg)
 fmt.Println("connected")
 defer ses.Close()
 checkErr(err)
 fmt.Println("Session PrefetchRowCount :", ses.Cfg().StmtCfg.PrefetchRowCount())
 err = ses.Cfg().StmtCfg.SetPrefetchRowCount(uint32(1000))
 checkErr(err)
 fmt.Println("Session PrefetchRowCount :", ses.Cfg().StmtCfg.PrefetchRowCount())

 stmtTbl, err := ses.Prep(
  `declare
  l_sql varchar2(32767);
  l_cnt pls_integer;
begin
  l_sql := 'drop TABLE emp_go purge';
  select count(*) into l_cnt from user_tables where table_name='EMP_GO';
  if l_cnt > 0 then
    execute immediate l_sql;
  end if;
end;`)
 defer stmtTbl.Close()
 checkErr(err)
 rowsAffected, err := stmtTbl.Exe()
 checkErr(err)
 fmt.Println(rowsAffected, " rows Affected. drop table emp_go if exists.")

 rowsAffected, err = ses.PrepAndExe("CREATE TABLE emp_go (empno number(5,0), ename VARCHAR2(50))")
 checkErr(err)
 fmt.Println("table emp_go created")

 rowsAffected, err = ses.PrepAndExe("ALTER SESSION SET TRACEFILE_IDENTIFIER='go1'")
 checkErr(err)
 //SELECT VALUE FROM V$DIAG_INFO WHERE NAME = 'Default Trace File';
 rowsAffected, err = ses.PrepAndExe("begin dbms_monitor.session_trace_enable( WAITS=>TRUE, binds=>true); end;")
 checkErr(err)

 tx1, err := ses.StartTx()
 stmt, err := ses.Prep("INSERT INTO emp_go (empno, ename) VALUES (:N1, :C1)")
 defer stmt.Close()
 checkErr(err)

 fmt.Println("Demo bulk/batch insert. Bulk DML with slice (array in Go)")
 l_no := make([]int64, 4)
 l_name := make([]ora.String, 4)

 for n, _ := range l_no {
  l_no[n] = int64(n)
  l_name[n] = ora.String{Value: fmt.Sprintf("Mr. %v", n)}
  fmt.Println(n)
 }
 rowsAffected, err = stmt.Exe(l_no, l_name)
 checkErr(err)
 fmt.Println(rowsAffected, " rows add.")
 tx1.Commit()
 fmt.Println("commit")

 fmt.Println("Demo fetch records")
 stmtQry, err := ses.Prep("SELECT /* set PrefetchRowCount = 0 */ empno, ename FROM emp_go")
 defer stmtQry.Close()
 checkErr(err)
 err = stmtQry.Cfg().SetPrefetchRowCount(uint32(0))
 checkErr(err)

 fmt.Println("stmtQry.Cfg().PrefetchRowCount default:", stmtQry.Cfg().PrefetchRowCount())
 rset, err := stmtQry.Qry()
 checkErr(err)

 for rset.Next() {
  fmt.Println(rset.Row[0], rset.Row[1])
 }
 checkErr(rset.Err)

 stmtQry, err = ses.Prep("SELECT /* set PrefetchRowCount = 200 */ empno, ename FROM emp_go")
 defer stmtQry.Close()
 checkErr(err)
 err = stmtQry.Cfg().SetPrefetchRowCount(uint32(200))
 checkErr(err)
 fmt.Println("stmtQry.Cfg().SetPrefetchRowCount(200)", stmtQry.Cfg().PrefetchRowCount())

 rset, err = stmtQry.Qry()
 checkErr(err)

 for rset.Next() {
  fmt.Println(rset.Row[0], rset.Row[1])
 }
 checkErr(rset.Err)

 rowsAffected, err = ses.PrepAndExe("begin dbms_monitor.session_trace_disable(); end;")
 checkErr(err)

 rset, err = ses.PrepAndQry("SELECT VALUE FROM V$DIAG_INFO WHERE NAME = 'Default Trace File'")
 for rset.Next() {
  fmt.Println(rset.Row[0])
 }

}


func checkErr(err error) {
 if err != nil {
  panic(err)
 }
}


Output:

Session PrefetchRowCount : 0
connected
Session PrefetchRowCount : 2000
Session PrefetchRowCount : 1000
0  rows Affected. drop table emp_go if exists.
table emp_go created
Demo bulk/batch insert. Bulk DML with slice (array in Go)
0
1
2
3
4  rows add.
commit
Demo fetch records
stmtQry.Cfg().PrefetchRowCount default: 0
0 Mr. 0
1 Mr. 1
2 Mr. 2
3 Mr. 3
stmtQry.Cfg().SetPrefetchRowCount(200) 200
0 Mr. 0
1 Mr. 1
2 Mr. 2
3 Mr. 3
/home/oracle/app/oracle/diag/rdbms/cdb1/cdb1/trace/cdb1_ora_17092_go1.trc

.
SQL session trace:

tkprof /home/oracle/app/oracle/diag/rdbms/cdb1/cdb1/trace/cdb1_ora_17092_go1.trc rpt.txt
.
TKProf report:
.
Before,  PrefetchRowCount = 0.

SELECT /* set PrefetchRowCount = 0 */ empno, ename
FROM emp_go


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          1          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        3      0.00       0.00          0          9          0           4
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        5      0.00       0.00          0         10          0           4

.
After SetPrefetchRowCount = 200.
.
SELECT /* set PrefetchRowCount = 200 */ empno, ename
FROM emp_go


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          1          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        1      0.00       0.00          0          7          0           4
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        3      0.00       0.00          0          8          0           4

.
Bulk INSERT,
.
INSERT INTO emp_go (empno, ename)
VALUES (:N1, :C1)


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0        
     4         43           4
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        2      0.00       0.00        


Reference:
.
Oracle driver basic, http://mujiang.blogspot.com/2015/12/golang-connect-to-oracle-database.html
driver readme, https://github.com/rana/ora/blob/master/README.md
Performance, http://docs.oracle.com/database/121/JJDBC/oraperf.htm
06:32 Golang Exercise: Images (2245 Bytes) » Developer 木匠
code:

package main

import "golang.org/x/tour/pic"
import "image"
import "image/color"


type Image struct{}

func (p Image) Bounds() image.Rectangle {
 return image.Rect(0, 0, 256, 256)
}


func (p Image) ColorModel() color.Model {
 return color.RGBAModel
}


func (m Image) At(x, y int) color.Color {
 v := uint8(x * y)
 return color.RGBA{v, v, 255, 255}
}


func main() {
 m := &Image{}
 pic.ShowImage(m)
}


Ref:

http://tour.golang.org/methods/16
06:32 Golang Exercise: HTTP Handlers (2938 Bytes) » Developer 木匠
Code:

package main

import (
 "fmt"
 "log"
 "net/http"
)


type String string

func (h String) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 fmt.Fprint(w, h)
}


type Struct struct {
    Greeting string
    Punct    string
    Who      string
}


func (h *Struct) ServeHTTP(w http.ResponseWriter, r*http.Request){
 fmt.Fprintf(w, "%v %v %v", h.Greeting, h.Punct, h.Who)
}



func main() {
 // your http.Handle calls here
 http.Handle("/string", String("I'm a good man."))
 http.Handle("/struct", &Struct{"Hello", ":", "Gophers! 你好吗?"})
 log.Fatal(http.ListenAndServe("localhost:4000", nil))
}



Call example:

http://localhost:4000/struct

http://localhost:4000/string

Reference:

http://tour.golang.org/methods/14


06:32 Golang Exercise: Equivalent Binary Trees (3709 Bytes) » Developer 木匠
code:

package main

import "golang.org/x/tour/tree"
import "fmt"


// Walk walks the tree t sending all values
// from the tree to the channel ch.
func Walk(t *tree.Tree, ch chan int) {
 if t == nil {
  close(ch)
  return
 }
 if t.Left != nil {
  Walk(t.Left, ch)
 }
 ch <- t.Value
 //fmt.Println("send ", t.Value)
 if t.Right != nil {
  Walk(t.Right, ch)
 }
}


// Same determines whether the trees
// t1 and t2 contain the same values.
func Same(t1, t2 *tree.Tree) (b_same bool) {
 c1 := make(chan int)
 c2 := make(chan int)
 b_same = true
 go Walk(t1, c1)
 go Walk(t2, c2)
 for i := 0; i < 10; i++ {
  //fmt.Println("get ", <-c1, <-c2)
  if <-c1 != <-c2 {
   b_same = false
   break
  }
 }
 return
}


func main() {
 //t1 := tree.New(1)
 //fmt.Println(t1)
 fmt.Println(Same(tree.New(1), tree.New(1)))
 fmt.Println(Same(tree.New(1), tree.New(2)))
}

 
Output:

true
false

Program exited.
 
Function Same, version 2:
 
func Same(t1, t2 *tree.Tree) (b_same bool) {
 c1 := make(chan int)
 c2 := make(chan int)
 b_same = true
 go func() {
  Walk(t1, c1)
  close(c1)
 }()
 go func() {
  Walk(t2, c2)
  close(c2)
 }()
 for {
  v1, ok1 := <-c1
  v2, ok2 := <-c2
  if !ok1 || !ok2 {
   return ok1 == ok2
  }
  if v1 != v2 {
   b_same = false
   break
  }
 }
 return
}
Ref:
http://tour.golang.org/concurrency/8
https://golang.org/doc/play/tree.go
https://github.com/golang/tour/blob/master/tree/tree.go
06:32 From Python chapter 2 code solution (3768 Bytes) » Developer 木匠
Here is my answer for book:

从Python开始学编程


Chapter 2, Appendix A, Exam.

https://book.douban.com/subject/26919485/
..

from itertools import repeat

def gen():
int_rate = [0.01, 0.02, 0.03, 0.035, 0.05]
for r in int_rate:
yield r
for r in repeat(0.05):
yield r

def num_year_pay() :
house_price = 500000
year_pay = 30000
year_n = 1
rem = house_price

print(year_pay)

for r in gen():
if rem <=0:
break
print(r)
rem = rem * (1 + r) - year_pay
year_n = year_n + 1
print(rem)

return year_n

if __name__ == "__main__":
print("Number of years to pay loan: ", num_year_pay())

Number of years to pay loan: 31

06:32 Complete spark 2.2.0 installation guide on windows 10. (23601 Bytes) » Developer 木匠
Goal
----
The step by step Apache Spark 2.2.0 installation guide on windows 10.


Steps
----
Download winutils.exe from git, https://github.com/steveloughran/winutils
E.g.: https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/winutils.exe

mkdir /tmp/hive

winutils.exe chmod -R 777 E:\tmp\hive
or
winutils.exe chmod -R 777 /tmp/hive

set HADOOP_HOME=E:\movie\spark\hadoop
mkdir %HADOOP_HOME%\bin
copy winutils.exe to %HADOOP_HOME%\bin

Download Spark: spark-2.2.0-bin-hadoop2.7.tgz from http://spark.apache.org/downloads.html
cd E:\movie\spark\
# in MINGW64 (git / cywin)
tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz
# or use 7-Zip

cd E:\movie\spark\spark-2.2.0-bin-hadoop2.7
bin\pyspark.cmd



Notes
----
%HADOOP_HOME%\bin\winutils.exe must be locatable.

Folder "E:\movie\spark\hadoop" is just an example, it can be any folder.

Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+.
http://spark.apache.org/docs/latest/

Reference
----
https://wiki.apache.org/hadoop/WindowsProblems


Here is the example output when start pyspark successfully:

E:\movie\spark\spark-2.2.0-bin-hadoop2.7>bin\pyspark.cmd

Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 18:41:36) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/11/17 19:07:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/17 19:07:36 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.2.0
      /_/

Using Python version 3.6.1 (v3.6.1:69c0db5, Mar 21 2017 18:41:36)
SparkSession available as 'spark'.
>>>
>>> textFile = spark.read.text("README.md")
17/11/17 19:08:03 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes
>>> textFile.count()
103

>>> textFile.select(explode(split(textFile.value, "\s+")).name("word")).groupBy("word").count().show()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'explode' is not defined

>>> from pyspark.sql.functions import *
>>> textFile.select(explode(split(textFile.value, "\s+")).name("word")).groupBy("word").count().show()
+--------------------+-----+
|                word|count|
+--------------------+-----+
|              online|    1|
|              graphs|    1|
|          ["Parallel|    1|
|          ["Building|    1|
|              thread|    1|
|       documentation|    3|
|            command,|    2|
|         abbreviated|    1|
|            overview|    1|
|                rich|    1|
|                 set|    2|
|         -DskipTests|    1|
|                name|    1|
|page](http://spar...|    1|
|        ["Specifying|    1|
|              stream|    1|
|                run:|    1|
|                 not|    1|
|            programs|    2|
|               tests|    2|
+--------------------+-----+
only showing top 20 rows

>>>

from pyspark.sql.functions import *


# module for pyspark,
from pyspark.sql import SparkSession
from pyspark.sql.window import Window
from pyspark.sql.functions import *
from pyspark.sql import *

Employee = Row("empno", "ename", "job", "mgr", "hiredate", "sal", "comm", "deptno")

emp1 = Employee(7369, "SMITH", "CLERK", 7902, "17-Dec-80", 800, 20, 10)
emp2 = Employee(7876, "ADAMS", "CLERK", 7788, "23-May-87", 1100, 0, 20)

df1 = sqlContext.createDataFrame([emp1, emp2])

sparkSession = SparkSession.builder.master("local").appName("Window Function").getOrCreate()

empDF = sparkSession.createDataFrame([
      Employee(7369, "SMITH", "CLERK", 7902, "17-Dec-80", 800, 20, 10),
      Employee(7499, "ALLEN", "SALESMAN", 7698, "20-Feb-81", 1600, 300, 30),
      Employee(7521, "WARD", "SALESMAN", 7698, "22-Feb-81", 1250, 500, 30),
      Employee(7566, "JONES", "MANAGER", 7839, "2-Apr-81", 2975, 0, 20),
      Employee(7654, "MARTIN", "SALESMAN", 7698, "28-Sep-81", 1250, 1400, 30),
      Employee(7698, "BLAKE", "MANAGER", 7839, "1-May-81", 2850, 0, 30),
      Employee(7782, "CLARK", "MANAGER", 7839, "9-Jun-81", 2450, 0, 10),
      Employee(7788, "SCOTT", "ANALYST", 7566, "19-Apr-87", 3000, 0, 20),
      Employee(7839, "KING", "PRESIDENT", 0, "17-Nov-81", 5000, 0, 10),
      Employee(7844, "TURNER", "SALESMAN", 7698, "8-Sep-81", 1500, 0, 30),
      Employee(7876, "ADAMS", "CLERK", 7788, "23-May-87", 1100, 0, 20)
])


partitionWindow = Window.partitionBy("deptno").orderBy(desc("empno"))
sumTest = sum("sal").over(partitionWindow)
empDF.select("*", sumTest.name("PartSum")).show()

partitionWindowRow = Window.partitionBy("deptno").orderBy(desc("sal")).rowsBetween(-1, 1)
sumTest = sum("sal").over(partitionWindowRow)
empDF.select("*", sumTest.name("PartSum")).orderBy("deptno").show()

06:32 Best Time to Buy and Sell Stock series. (6472 Bytes) » Developer 木匠
股票收益最大化,一系列有五六个题目。

这里仅回顾我前进中的心路, 停下来, 做个小结。
未来两个月后,下一步,我会重新训练解题思路。

正文开始了,

Q: 允许交易两次的股票收益最大化,这道题是不是枚举/模拟类题目?
Q: 这个题我自己没有想出答案。有没有什么指南训练一下,可以提高解题水平?

我用的是两端扫描的办法,从左和从右扫描得到,每端的依次的最大收益值,然后再扫描一次,两边相加求最大值,时间复杂度还是o(n)。
A:左边和右边加起来,找到全局最大化。

去年夏天,第一次面试前端,问了普通数字和罗马数字转换,我就没有想出解题思路。
普通数字和罗马数字转换,应该是标准的模拟/枚举题目。

2. Best Time to Buy and Sell Stock with Transaction Fee

这道题我毫无头绪。卡住两三天。
这一类题目,该怎么学习和训练呢?

大师点拨:
A:
This is a DP and Greedy related problem, I don't think you learn the algorithm yet.
It will be taught in advanced algorithm jiuzhang class. I will suggest you to skip.

At this moment, you are not the scientist who can invent the algorithm.
In fact almost all people are NOT. This is why...

😇 谢谢分析和打气鼓励。

据说最好的学习策略,就是不断的取得小胜利,我在这个问题上两三天取得了一个大失败,觉得很受挫折。
猛一看像是个简单的小问题,就想给他一举拿下。
这是九章算法班,最后一节课前缀和知识点,引出了一个股票交易问题,没想到延伸出了五六个问题,整个做法思路都变样了。
特别是这一道题,facebook考过,我就想着应该做出来。也算一个准备。

我的直观解法,通过了一多半测试,总是不甘心。
谢谢大师点拨,让我从死胡同里解脱出来,摆正心态,继续前进。

附录:

The hardest problem in this series.

393. Best Time to Buy and Sell Stock IV
lc, https://www.lintcode.com/problem/best-time-to-buy-and-sell-stock-iv

constraint: trade k times, get max gain.

idea:

我自己都没有搞清楚是怎么通过测试验收的. 虽然代码是我从头写出来的。

essential code:

    def maxProfit(self, k, prices):
        if not prices or len(prices) == 1:
            return 0
        buy = -prices[0]
        sell = 0
        if k > len(prices):
            for price in prices:
                buy = max(buy, sell - price)  # core code 
                sell = max(sell, buy + price)  # core code 
            return sell
        buys = [-float("inf")] * k
        sells = [0] * k
        for price in prices:
            buys[0] = max(buys[0], -price)
            for i in range(k):
                if i > 0:
                    buys[i] = max(buys[i], sells[i - 1] - price)
                sells[i] = max(sells[i], buys[i] + price)
        return sells[k - 1]
06:32 2018 first mock coding interview (2047 Bytes) » Developer 木匠
周五进行了一场模拟面试,S是应聘者,咱做考官。

首先说我自己,我一直就没有进入角色,都不知道怎么开展技术面试,我 2015年 在德银面试委员会待过一段时间,面试了40多个候选人,主要就问软技能的问题。那时候我还不会写程序。

下面是我面试过程中的一些感想和议论。

我觉得有一个手写版比较好,像九章令狐冲老师用的,讲解题目,解释原理和概念,很方便,不知道哪里可以买到 价廉物美的先试用?

战友 S,沟通能力特别强,始终都能把自己的意思清楚的表达出来,就是说话的时间多了,写题的时间就少了。

九章老师关于沟通的时间分配是这样讲的:

• 开始,问清楚题目的需求,确保自己完全理解需求,不要主观臆断,然后说出自己的解决方案的思路,征求考官的认可,也可以说出多个方案跟考官商议,共同确定一个较好的实现方法。要点是把考官当成自己未来的同事,你跟同事沟通的时候也要简洁明了,既要说清楚,又不能过于繁琐。
• 然后就让考官去玩手机,自己闷头做题。
• 做完题以后再跟考官简短说个总结就可以了,如果还有时间,可以解释一下代码中比较难以理解的部分。

S同学 分析能力强,思路清晰。

下面这些建议说给我,供给战友们参考。

各种常用基本算法和数据结构 需要多练习,直到熟练掌握,倒背如流。

多学习各种常用的优质模版代码,简短而易读,不容易出错,易于诊断。
代码长了,逻辑复杂,就容易出错。
多仿照练习优质模版代码,4遍5遍,直到熟练掌握。

S同学 大有可为,我们一线大厂见。

再加一句:  S同学的英语比我好多了,阴阳顿挫,发音清晰。

06:32 运维的经济分析 (11794 Bytes) » 知道分子
运维承担着网站生产环境及相关设备、基础设施的运行维护工作,基础设施及各种设备、软件的评估、选择、采购、分配、部署、运用、维护都在运维,相应地,每年技术预算中大部分资本性支出(CAPEX)和运营性支出(OPEX)也在运维发生。因此,随着网站规模日益扩大,生产环境日趋复杂,如何科学分析数据,合理配置资源,在保持稳定、安全的前提下,追求紧凑、高效的发展,已成为亟待研究的课题。

运维的经济分析,旨在从经济学视角观察运维过程中的种种现象,运用经济学方法进行量化比较分析,找出优化改善的途径,节约成本、提高效率。范围涵盖服务器利用率、数据中心利用率、报警有效性、网络流量分层、访问均匀度、解决方案替代性、流程规范的经济性、应用项目运维成本费用划分等多个研究方向。

1. 服务器利用率

服务器的利用率,是指服务器内各项部件性能容量的综合利用率。服务器各项部件的配置,应当与应用对各部件的实际需求比例相匹配,给 CPU 密集型应用配置超量高速内存,给内存密集型应用配置高频强劲 CPU,都会形成不必要的浪费。此外,应用集群应按照同样的标准配备冗余服务器,过低的冗余会影响安全,过度的冗余则会产生浪费。该研究方向的目标,是根据长期追踪统计分析结果,得出应用集群利用率模型,确定相应的服务器(或虚拟机)最优配置和冗余度标准。

2. 数据中心利用率

数据中心利用率,是指数据中心空间和能源的利用率。数据中心地面和机架空间的优化配置,机架内服务器单位空间、能耗产出效益,都应作定量比较分析。旧服务器应该依据何种标准淘汰以及如何替换,选择哪一种服务器更能充分利用机架空间,能耗受限条件下选择哪一种服务器可以更充分利用能源,数据中心内部如何科学划分区域以满足设备潜在的扩展需求,不同数据中心之间如何合理分配部署各种不同特点的应用和设备,都是该方向需要研究的内容。

3. 报警有效性

随着监控点的日益增长和逐步细化,由于不合理的阈值、过于敏感的检查调度策略、过于频繁的消息传递所产生的无效报警越来越多,干扰正常的工作和生活,产生不必要的通讯费用和人力资源浪费。应合理定义报警,分配不同的报警手段,结合应用关联关系进行报警洪水抑制,消除多报、少报、错报、漏报,提高报警有效性,节约通讯费用和人力成本。

4. 网络流量分层

网络流量由多种成分和层次构成,除了内部外部、上行下行之分外,还可以根据不同目的、价值分为正常访问、搜索爬虫、运营推广、自制爬虫、扫描攻击等多个层次的流量,每个层次内部还可以进一步细分。正常访问应当得到优先保证,特别是其中的交易相关访问。搜索爬虫则应给予区别对待,允许爬抓的目录应合理控制以免影响性能,非主流搜索爬虫应根据成本收益分析结果限制访问,行为恶劣的爬虫应当禁止访问。运营推广流量要提前评估,安排合理的时间段,在运营收益和运维成本之间求平衡。自制爬虫和扫描攻击应当禁止。通过网络流量分层,抵御恶意流量、消除垃圾流量、限制低价值流量、保护高价值流量,并且通过合理调度,优化运营推广流量的成本收益率。

5. 访问均匀度

网站的访问量趋势往往是一条高低起伏的曲线,面向特定时区的网站尤为明显。大多数ISP和CDN都是按总体的95%百分位数流量收取费用,实际上网站的路由器、交换机、服务器、存储资源都是根据访问量峰值再加上一定冗余度配置,可以说,访问量的峰值决定了成本和费用支出水平。但访问量本身并不是铁板一块,在网络流量分层的基础上,还可以进一步分析不合理的访问,比如大图小用、静态文件过期时间过短、不必要的Cookie上传、高峰期执行计划任务等等,通过错时避峰、削峰填谷等手段,让访问变得更均匀,在节约成本费用的同时,使资源得到充分利用。

6. 解决方案替代性

SSD替代机械硬盘、刀片/多板服务器替代传统机架式服务器、PC+MySQL替代小型机+Oracle、分布式存储替代集中式存储、开源负载均衡软件替代商业专用设备、第三方服务商A替代第三方服务商B,替代随时都会发生。问题在于,能在哪些/个场景中替代,能在多大程度上替代,由此产生的成本和收益比例如何,有没有潜在风险,综合来看是否经济。在各种替代解决方案的评估阶段,应充分挖掘各个维度的数据,科学论证替代性,提供决策参考。

7. 流程规范的经济性

流程规范是双刃剑,合理的流程规范可以提高效率、降低风险,不合理的流程规范则降低效率、提高成本。为了规避哪些风险,相应地建立哪些流程规范,这些流程规范的实施需要多少成本,会不会降低效率,应进行科学评估。除了流程规范本身的经济性分析之外,还要对流程规范具体实施方式的经济性作进一步分析评估。清理、修订、简化那些繁冗的、成本收益率较低的、操作复杂不易遵从的,甚至有副作用的不合理流程规范,保留和建立必要、精简、紧凑的流程规范。

8. 应用项目运维成本费用划分

各个应用项目的成本费用都在大部门预算中合并,导致负责具体应用的小团队无法获知所负责应用究竟有多少固定资产投入,每月每季每年产生多少运维费用,相互之间也无从比较。尤其是对于直接产生收益的项目,缺少细分的成本费用数据,就无法计算成本收益率,从而可能影响经营决策。公共部门的各项服务如果无法定价,就会被滥用而导致无效率,运维作为技术部资源最集中的公共部门,应当对各项资源、服务定价计费,按应用/项目准确分摊成本费用,建立横向对比参照体系,让应用/项目团队对所耗用的成本费用负责,成为具有经济理性的主体,在此基础上,将来可以更多利用经济手段而非行政命令进行运维规划。
06:32 浅尝 Privacy: A Very Short Introduction (8506 Bytes) » 知道分子

Boston Legal S02E13 Too Much Information 里讲了这么一个案子,原告小女孩的母亲患有精神疾病,她父亲使用妻子的社会安全号码和出生日期登录健康管理公司(HMO) Well Benefits 网站,查到妻子的健康档案,内容包括定期诊疗的时间、精神科医生姓名、办公室地址等,然后前往医生办公室门外守候,将妻子杀死。小女孩于是向法院提起民事诉讼,状告 HMO Well Benefits,要求赔偿因网站泄露隐私导致其母被其父杀害的损失。

被告辩称,其一,原告的父亲杀害母亲,系出于主观犯罪意图,并非 Well Benefits 之过;其二,Well Benefits 网站符合州与联邦互联网安全相关规定标准,已尽到安全保障责任;其三,Well Benefits 公司愿意向小女孩支付两万美元作为慰问礼金。如果陪审团判 Well Benefits 承担赔偿责任,那么这个判决将打开法律诉讼的潘多拉魔盒,不仅是针对 HMO,还会把所有互联网公司都拖下水。

著名律师 Alan Shore 却认为,把客户的个人私密信息放在网站上,应当是可以预见的危险,网站理应履行相应的安全保密义务,承担因疏忽大意导致客户隐私泄露而产生的损害赔偿责任。使用 5 位邮政编码、性别和生日条件组合,就能唯一辨识 87% 的美国人口,隐私原本就是这么脆弱。Well Benefits 网站在本案中为谋杀行为提供了客观上的便利,只需利用被害人的社会安全号码和生日就能访问关键隐私信息,显然应受惩罚。

剧中陪审团最后判原告胜诉,被告须向原告支付 95 万美元补偿性赔偿金,以及 200 万美元惩罚性赔偿金。观众皆大欢喜。据说 Boston Legal 中的案件皆有所本,但不知这桩案子对应的是哪一个判例。

从前只要锁好门窗、拉上窗帘、藏入保险箱就能妥善保护的隐私,随着技术发展进步,网络渗透人们工作生活的一切场所,其泄露变得越来越难以避免。我们的户籍资料、健康档案、财务信息、金融资产、保险记录、联络通讯、兴趣爱好、旅游行程等隐私数据都保存在各种不同的网络和数据库中,一般只需要输入两三个字符串就能访问到这些信息。难以控制的是,除了我们自己之外,还有不少人可以从其它合法或非法渠道获取我们的隐私,并且随意利用,否则我们就不会在每年车险到期之前接听大量推销电话,就不会在宝宝出生之后收到各种奶粉、幼教广告。

此外,还有无处不在的监控手段,随时随地采集我们的隐私信息。比如,CCTV 闭路电视监控,方便抓捕罪犯的同时,也大量摄录了我们的样貌和行为;GPS 全球定位系统,仙人指路,但同时可以用来追踪你的行迹;RFID 射频识别标签,方便管理货物,但也可以用来做个人信息和货物信息的聚合研究;Cookies,网站将它植入你的浏览器,用于跟踪你的访问行为,分析你的个人偏好;搭线窃听,记录你的通讯对象和内容,等等。

That is how vulnerable we are. How vulnerable you are.
                                            -- Alan Shore, Boston Legal S02E13

是为第一章。
06:32 没有药方的病历 (9806 Bytes) » 知道分子

三个月前,我发现穿了一年多的 Nike 运动鞋边上有点破损,于是便开始寻找一双新鞋以备替换。这双深灰色的 Nike 是我穿过最舒服的鞋,轻巧透气、避震良好,正如鞋上标示的 Lunar Lite 字样,穿着它走路跑步就像宇航员在月球表面弹跳那样轻松自如。可惜翻遍网上线下的 Nike 专卖店,再也找不到合意的新鞋,同一系列的新款外观太艳,不适合我这样的中年人,其它系列的要不是外观有问题就是脚感不舒服。另外品牌的鞋也试过,更没有可与这双旧鞋相匹敌的。

这双鞋在美国研发,越南生产,中国销售。100多美元的售价,很有可能其中55美元归美国 Nike,20美元归越南生产者和原料供应商,25美元归中国的经销商和物流环节。对于志在“颠覆品牌全球统治”的文化左派来说,这是个很好的题目,跨国公司 Nike 尽享暴利、越南血汗工厂、中国......但对我而言,这只是一双外观稳重、脚感舒适的鞋,只恨当时没有买两双,否则现在也不必为找新鞋挠头了。至于 Nike,我从小到大至少穿过十几双,试穿其它品牌款式的都不太适应,就一直认这个牌子买而已。

NO LOGO 这本书介绍了很多大公司作恶的事。比如英国药厂 Boots 赞助加州大学旧金山分校的 Betty Dong 博士,研究比较 Boots 生产的甲状腺药物 Synthroid 和某个无名药品之间的差异,只要能证明 Synthroid 比其它同类药物疗效更出色,那么就得到著名大学学者的背书,有望提升销量了。结果董博士的研究结果表明情况恰好相反,这两种药物从生物学的观点来看是完全相同的,服用名牌药品的 800 万美国人一年可以省下 3.65 亿美元。药厂和学校迅即联手阻止了董博士这篇研究报告刊出,直到最后被《华尔街日报》捅出来,还是晚了两年才得以发表。(p.124)还有科学家发现某药品具有副作用,甚至可能致人死亡,研究结果不仅被药厂阻止发表,还差点丢了教职。某医生受纺织厂委托调查肺病案例,发现该厂肺病比例奇高,纺织厂和医院联手阻止医生发表报告,并且关闭了他做研究的诊所。(p.125)

从来都有黑心无良的商家、机构和组织,否则还要媒体、警察和法院干吗?如果把这些问题都归结于“品牌”之罪,并且幻化出所谓的“品牌全球统治”,未免瞄错了靶子。三聚氰胺事件之后,三鹿品牌倒了,伊利蒙牛不敢喝了。二恶烷事件之后,霸王洗发水一蹶不振。化工原料做紫砂锅,美的品牌一落千丈。还有以前肯德基掺苏丹红、金华火腿抹敌敌畏、太仓肉松用死猪肉+老母猪肉+豆粉+双氧水制成,舆论媒体揭露之后,这一系列的品牌都遭了现世报应。哪里有什么“品牌全球统治”可言?

品牌固然能带来更大的销量、更高额的利润,但是塑造和维护一个品牌谈何容易。创新研发、外观设计、产品质量、客户服务、绿色环保、社会责任,每一个环节都不能松懈。现代社会信息流通速度何其迅捷,同行竞争何其激烈,只要被竞争对手和大众媒体嗅到一点点腥味,任你多大牌都得吃不了兜着走。体育用品公司赞助体育赛事、赞助学校和社区的体育设施,房地产公司为乡村小学修建清洁厕所,电脑公司为公立学校免费提供软硬件设备,企业如此树立品牌,对社会有益无害。

这是一份没有药方的病历,罗列出一大堆“症状”,都记在抽象病人“品牌”的名下。其实这些病症各有所属,自从商品诞生以来,人性中的恶就周而复始地开花结果。没错,商业社会需要更开放的舆论、更有力的法治来监管和维系,不要指望无良商家/机构/组织会自行纠正错误,但也不能像本书作者那样幼稚地认为,通过所谓“文化反堵”、“收复街道”、“群情激奋”就能“颠覆品牌全球统治”,何况她想要“颠覆”的本是一架不存在的风车。


(《NO LOGO:颠覆品牌全球统治》,【加】Naomi Klein 著,徐诗思 译,广西师范大学出版社 2009年5月出版)

**  零售价299美元的 iPod,其中80美元归苹果公司,东芝微硬盘73美元,显示模块20美元,多媒体处理芯片8美元,控制器芯片5美元,中国组装厂仅得4美元。 Hal R. Varian, An iPod Has Global Value. Ask the (Many) Countries That Make It.  New York Times 2007-06-28
06:32 大法官说了算 (14715 Bytes) » 知道分子

如果说起美国联邦最高法院,还只记得那几个千年老案,“马伯里诉麦迪逊”、“米兰达诉亚利桑那州”、“罗伊诉韦德”等等,那你就 out 了,好像改革开放三十年来美国高院没干什么正事似的。

作为三权分立的一支,美国高院始终维护着宪法的权威,通过不断修正国家法治原则、界定公民宪法权利,以客观公正的判决赢得人民的信任与认可。进入新世纪以来的十年,随着日新月异的科技进步、文化发展,社会生活发生了很大变化,由此也产生了各种前所未有的新问题,纷纷呈到大法官们案前。两百多年前制宪先贤写在纸上的字句,应当如何解释并适用于显示屏上闪烁的疑难杂症呢?大法官说了算。

本书作者何帆是最高人民法院的一位年轻法官,曾当过四年警察,后一路进修至人民大学刑法学博士。活跃于网络,网名“盒饭”。勤于著译,写过小说《一个伪知识分子的警察生涯》,也写过学术专著《刑事没收研究:国际法与比较法的视野》、《刑民交叉案件审理的基本思路》,还翻译过畅销书《作为法律史学家的狄更斯》、《九人:美国最高法院风云》等。《大法官说了算》,据其供述,是在译了《九人》人之后,为缓解“译书综合症”而进行自我治疗的病程记录。(p.312)

去年4月译完《九人》,5月苏特大法官突然宣布辞职,作者就感觉“演出开始了”,“因为随之展开的,将是民主党总统15年来第一次大法官提名、驴象两党新一轮政治角力、最高法院两种意识形态的此消彼长……这些连锁效应,是《九人》故事的现实延伸,更是治愈‘译书综合症’的灵丹妙药”(同上)。这一年来,奥巴马提名西班牙裔女大法官索尼娅·索托马约尔上任接替苏特大法官,随后史蒂文斯大法官宣布退休,又提名原哈佛法学院院长、美国联邦政府首席律师埃琳娜·卡根接替,最高法院大法官的性别构成、力量对比新格局给人以无限想象。所以,承前书之余绪,对美国最高法院运作机制及近十年来的典型判例作一番研究总结,是个不错的选题。

大法官说了算,是在宪法没有明确规定的地方说了算。当年制宪先贤“为了争取更多州批准宪法,故意搁置重大争议,对许多条款语焉不详,因此,最后形成的宪法条文中,只含糊规定司法权属于最高法院及国会设立的下级法院,而对联邦最高法院的组成、运作、权限,并未做过多交代”,最高法院法官称谓、审案方式、判决机制、司法权和立法、行政权之间的具体制衡方式,制宪者都没有说清楚。(p.3)所以,“规矩没有自己定”。马歇尔大法官曾推动创立最高法院内部规则,如果全体法官意见一致,则由一人代表全体撰写“法院意见”(Opinion of the Court);若有意见分歧,则由多数方、少数方各一人撰写“多数意见”(Majority Opinion)和“异议意见”(Dissenting Opinion);如果多数方某位大法官同意最终结论,但不同意判决理由,可以单独发布“协同意见”(Concurring Opinion)。最高法院每年开庭期持续9个月,夏季3个月为闭庭期,每年上诉至高院的案件数以千计(2009年收到7738起),由于时间有限,开庭期间选择哪些案子来审,也是大法官们说了算(2009年开庭审理87起)。“经过两百多年的运转,联邦最高法院已形成一套行之有效的规则与惯例,大部分都由大法官们自己实践、总结,具体细节也一直在不断调整。”(p.5)

大法官说了算,是在宪法审查/违宪审查方面说了算。最高法院既没有军权也没有财权,不能主动出击,本是三权中最弱的一支。在“马伯里诉麦迪逊”(Madbury v. Madison)案中,马歇尔大法官创造性地宣布,《司法法》第十三条因与宪法相抵触而无效,马伯里应先去下级法院起诉,由此为最高法院争来了一项超级武器:司法审查权。随后,马歇尔法院又乘胜追击,在“弗来彻诉佩克”(Flecther v. Peck)案中宣布最高法院对各州立法也有司法审查权。(p.7)拥有司法审查权,最高法院就可以推翻国会制定法案中的违宪条款,还可以审查政府部门的违宪施政。司法权与立法权、行政权之间的三权分立、相互制衡,至此才真正得以实现。罗伯特·杰克逊大法官在“西弗吉尼亚州教育委员会诉巴内特”(West Virginia State Board of Education v. Barnette,该案判决支持公民不向国旗致敬的自由)案的判决意见中写道:“如果在我们宪法的星空上有一颗不变的星辰,那就是,无论在政治、民族、宗教,还是其他舆论问题上,任何官员,不论其职位高低,都无权决定什么是正确的,也无权用言语或行动来强迫公民表达他们的信念。如果有什么情形允许这一例外,那么,我们现在决不允许它们发生!”(p.9)

大法官说了算,是在宪法条文的解释和具体适用方面说了算。2002年“明尼苏达州共和党诉怀特”(Republican Party of Minnesota v. White)案中,最高法院推翻联邦地方法院和上诉法院判决,多数方大法官认为,明尼苏达州1974年制定的法官选举规范中关于禁止法官候选人就争议性法律、政治议题公开发表看法的“宣示条款”限制了参选者的言论自由,违反宪法第一修正案,属无效条款。(p.99)2003年“美国诉美国图书馆协会”(United States v. American Library Association)案中,最高法院以6票对3票推翻联邦地方法院判决,多数方大法官认为《儿童互联网保护法》(Children's Internet Protection Act,简称 CIPA)关于在公共图书馆安装过滤软件的规定合宪。(p.106)2008年“巴泽诉里斯”(Baze v. Rees)案中,最高法院以7票对2票支持地方法院判决,裁定肯塔基州的注射死刑执行方式不违反宪法第八修正案,多数方大法官认为,“这是执行死刑,又不是医生会诊!”(p.185)2009年“联邦通讯委员会诉 Fox 电视台”(FCC v. Fox Television Stations)案中,最高法院支持 FCC 惩治“电台粗口”的新政策(p.108),在随后的 FCC 诉 CBS 案中,最高法院宣布撤销第三巡回上诉法院的判决,发回重审,要求下级法院领会前案判决精神,多数方大法官认为,连粗口都算不雅言论了,你这“瞬间露点”行为甭想蒙混过关。(p.112)2010年“公民联盟诉联邦选举委员会”(Citizens United v. FEC)案中,最高法院以5票对4票判决解除企业、工会对政治选举的资金介入,多数方大法官认为,宪法第一修正案的基础在于政府无权管制政治言论,而且宪法条文并没有规定言论自由只赋予个人,不给企业,因此,政府不得限制企业、工会“投资”选举宣传,更不能因为公民或公民组织发布政治言论就罚人家款,或者把人家扔到监狱里去。(p.125)

当然,在一个相互制衡的三权分立体系中,最高法院未必能绝对的“说了算”,何况其内部也存在着意识形态的不同阵营,大法官有时候也难免“算了说”、“说算了”、“算说了”。比如上面提到2010年“公民联盟诉联邦选举委员会”这桩案子,就引起奥巴马的强烈不满,判决当日即宣布将寻求两党参议员合作,以立法形式推翻这一判决,后来又在发表首次国情咨文时,当着台下6名大法官的面,公开谴责最高法院的该案判决。以至于阿利托大法官气得连连摇头叹息,还被现场记者拍个正着,罗伯茨首席大法官两个月后在阿拉巴马大学演讲时也表示,大法官们根本没必要去听什么国情咨文,总统当众攻击司法分支的行为非常糟糕。(p.219)又如,最高法院的财政预算掐在国会手里,罗伯茨首席大法官曾连续四年在美国联邦司法年度报告里摆事实讲道理,罗列出一大串数据说明法官薪酬下降导致司法质量下降的困境,要求国会给联邦法官加薪。然而国会一直对此置之不理,“在参议员们看来,六位数的年薪、终身制的岗位,外加退休后的足额薪水,联邦法官的待遇已远远超过了民选官员,有什么资格再要求提高待遇?”(p.215)所以在2009年的报告里,首席大法官干脆只写了短短几句:“时至今年,行政分支面临如此纷繁的棘手事务,许多同胞亦忙于应对时艰,公众或许更乐意读到一份言简意赅、要点突出的年度报告:法院系统运转良好,敬业的联邦法官们正尽心尽力履行职责。”(p.306)

《大法官说了算》文字清新可读,陈述事实不失专业,剖析理论不乏幽默,即便没有法学背景的普通读者,读完此书也能对美国司法系统及其最新发展有个比较完整的了解。书末罗列近 90 部中外参考文献,几乎涵盖大多数相关主要学术研究著作,足见作者用功之深,亦可供相关专业学生延伸阅读时参考。


(《大法官说了算——美国司法观察笔记》,何帆 著,法律出版社 2010 年 8 月出版)
06:32 国画 (7544 Bytes) » 知道分子

范宽:溪山行旅图


上周跟一群朋友吃饭,其中有一位新朋友是中国美院国画系研究生。仗着家里长辈和他们美院历代祖师多少有一点渊源,我便跟这位朋友攀谈套瓷,谈到国画学的前沿问题,气氛顿时凝重了起来。我想了解的是,国画这门艺术/技术/学术还有没有所谓的前沿,换言之,还有没有比较大的突破空间?

范宽的溪山行旅图,把山水画完了,若嫌不够,黄公望的富春山居图拿来作补充,其余没什么可看的了吧?众生相,张择端画完了。花鸟,吴昌硕画完了。虾,齐白石画完了。马,徐悲鸿画完了。牛,黄胄画完了。线条,前不久刚刚去世的吴冠中画完了。除了还没见过的外星人没画,奇形怪状的东西早就被八大山人画完了。国画,还有什么好画的?

黄灿然写过一篇震烁古今的论文,在两大传统的阴影下,读书杂志曾分两期刊出。文章说道,整个当代汉语写作都被笼罩在中国古典和西方古典两大传统的阴影下,其中尤为甚者,当代汉语诗歌面临着来自中国古典诗歌和西方现代诗歌(汉译)的巨大压力。比如登高这个题材,陈子昂的登幽州台歌、王之涣的登鹳雀楼,加上杜甫的登高和登岳阳楼,四首诗覆盖了时间与空间、历史与现实、悲观与乐观、近景与远景、静态与动态、个人身世与家国情怀,把登高写完了。

国画何尝不也面临着同样的问题?在中国画和西洋画两大传统的阴影下,现代人对于国画的鉴赏需求又不断萎缩,加之其它现代艺术和现代技术结合的产物纷纷推陈出新,共同形成了当代国画面临的压力和焦虑。国画囿于传统表达技法语言,本身已不可能有太大的突破,也许只有通过与其它对象结合、混搭,向其它艺术品输出中国画元素,才能实现自身存在的价值。

这位朋友认为,当代国画要追求传统艺术与时代精神的结合,说的就是这个意思吧。
06:32 mod_pagespeed:傻瓜式前端优化 (11119 Bytes) » 知道分子
曾经有位朋友自己做了个小网站,刚开始没什么人气,后来不小心把流量搞大了,用户抱怨访问缓慢,就让我帮忙看看哪里可以优化。那时 Steve Souders 老师的 YSlow 14条军规刚刚新鲜出炉,开宗明义第一句便是:网页性能 80% 消耗在前端。于是运用 Firefox+Firebug+YSlow 工具,轻松找到那些大图小用、过期时间太短、JS/CSS位置不正确、没有精简压缩的罪恶之源。前后只花一两周时间就把网站弄快了,好不得意。

今天 Google 发布的 mod_pagespeed for Apache 2,着实令人赞叹。原本需要一两周时间才能完成的前端优化工作,不到半个小时就能搞定,而且不用修改任何程序。现成的二进制安装包,下载来直接装入系统,自动添加配置,重启 Apache 之后,一切前端问题自动优化。CDN服务商 Cotendo 已在其CDN服务器上部署mod_pagespeed以加速客户网站访问,图片文件大小经自动压缩可减少20%-30%,页面加载时间最多可缩短50%。Go Daddy 也宣布将在其客户网站服务器上广泛部署mod_pagespeed。

mod_pagespeed 中有很多巧妙的设计,规避了以往必须要动复杂手术才能解决的问题。比如,它可以将图片文件的过期时间自动延长到一年,无论图片是否存在同名更新(即文件名不变,图片内容随时可能改变,从而无法设置较长的过期时间)。假设网站原来的logo图片引用和HTTP header如下:
HTML tag   : <img src="images/logo.gif" />
HTTP header: Cache-Control:public, max-age=300

经自动处理后,会变成:
HTML tag   : <img src="images/ce.c17941127d34679357baa1b36fb4ecc5.logo,g" />
HTTP header: Cache-Control:public, max-age=31536000

mod_pagespeed 把原来的 logo.gif 转化为名称唯一的 ce.c17941127d34679357baa1b36fb4ecc5.logo,g ,并且将原先的过期时间300秒延长到31536000秒(一年)。mod_pagespeed 则仍然以此前定义的 TTL 300 秒为周期,定期检查图片是否更新,一旦图片内容发生改变,文件名也会相应变化,如此就不用管同名更新的问题了。

又如最常见且最难以控制的“大图小用”(80x80的图片框里塞一张1024x768的高清墙纸),mod_pagespeed是这样处理的:
<img src="images/Puzzle.jpg" width="256" height="192" />

识别 IMG 标记中设置的宽度和高度值,自动缩放为相应大小和质量的图片:
<img src="images/ic.HASH.256x192xPuzzle,j.jpg" />

还有很多有趣的功能,在 http://www.modpagespeed.com 可以看到实例展示。

mod_pagespeed 显然很适合中小规模网站使用,不用太多费力于前端改造,就能迅速成倍提升客户体验。但对于大规模商业网站来说,我认为还是要慎重,因为它实时过滤处理每一次请求,存在不小的开销,而且作为beta版软件,即使要采用也得先进行充分测试。当然各种优化手段也可能有副作用,值得一提的是,mod_pagespeed 的文档写得不错,条件、限制、风险都很明确,建议详细研究后再作抉择。

06:32 Incast (7052 Bytes) » 知道分子
聊天益智,增广见闻,最近的业界八卦主题除了 Inception 之外还有 Incast。

话说现在基于 IP 的存储集群或分布式存储越来越普遍,数据分布在多台存储服务器上,客户端同时从多个节点同步访问数据。


卡内基梅隆大学并行数据实验室(PDL@CMU)的研究人员发现,当节点数扩张到一定程度时,会发生 TCP 流量崩溃。这就是 Incast 问题,数量渐进增长的多台存储服务器向一台客户机同时发送数据,可能导致交换机缓冲区溢出,数据包丢失,从而引发大量 TCP 包重传。TCP 重传超时(RTO,Retransmission TimeOut)最小值一般缺省设置为 200ms,在高带宽低延时的内部网络中,这样的设置让数据包要等待数十倍于传输的时间才能重传,因此网络带宽无法充分利用。


给交换机增加缓存容量是可行的一种方案,但是价格昂贵。简便易行的解决办法是降低 RTO 最小值,设置为 200μs,可以支持多达 47 个并发节点的满负荷传输。(Linux 2.6.28.10 内核补丁


详见: http://www.pdl.cmu.edu/Incast/
06:30 Will the Optimizer development team be at Oracle Open World? (670 Bytes) » Inside the Oracle Optimizer - Removing the black magic
The largest gathering of Oracle customers, partners, developers, and technology enthusiasts will happen in September when Oracle will host its annual user conference Open World in San Francisco and the Optimizer development group will be there! You will have two opportunities to meet the team -- attend the technical presentation "Inside the 11g Optimizer - Removing the mystery" on Tuesday morning at 9am or stop by the Oracle demo grounds (in Moscone West) to see all of the demos for the 11g new features and ask the development team any burning questions you may have!

06:30 Will the Optimizer Team be at Oracle Open World 2009? (1818 Bytes) » Inside the Oracle Optimizer - Removing the black magic
With only two and a half months to go until Oracle Open World in San Francisco, October 11-15th, we have gotten several requests asking if we plan to present any session at the conference.

We have two session and a demo station in the Database campground at this year's show. We will give a technical presentation on What to Expect from the Oracle Optimizer When Upgrading to Oracle Database 11g and the Oracle Optimizer Roundtable.

The technical session, which is on Tuesday Oct 13 at 2:30 pm, gives step by step instructions and detailed examples of how to use the new 11g features to ensure your upgrade goes smoothly and without any SQL plan regressions.

The roundtable, which is on Thursday Oct. 15th at 10:30 am, will give you a first hand opportunity to pose you burning Optimizer and statistics questions directly to a panel of our leading Optimizer developers. In fact if you plan to attend the roundtable and already know what questions you would like to ask, then please send them to us via email and we will be sure to include them. Other wise, you can hand in your questions at our demo station at any stage during the week, or as you enter the actual session. Just be sure to write your questions in clear block capitals!

We look forward to seeing you all at Oracle Open World.

06:30 Why do I have hundreds of child cursors when cursor_sharing set to similar in 10g (7765 Bytes) » Inside the Oracle Optimizer - Removing the black magic

Recently we received several questions regarding a usual situation where a SQL Statement has hundreds of child cursors. This is in fact the expected behavior when

  1. CURSOR_SHARING is set to similar

  2. Bind peeking is in use

  3. And a histogram is present on the column used in the where clause predicate of query

You must now be wondering why this is the expected behavior. In order to explain, let's step back and begin by explaining what CURSOR_SHARING actually does. CURSOR_SHARING was introduced to help relieve pressure put on the shared pool, specifically the cursor cache, from applications that use literal values rather than bind variables in their SQL statements. It achieves this by replacing the literal values with system generated bind variables thus reducing the number of (parent) cursors in the cursor cache. However, there is also a caveat or additional requirement on CURSOR_SHARING, which is that the use of system generated bind should not negatively affect the performance of the application. CURSOR_SHARING has three possible values: EXACT, SIMILAR, and FORCE. The table below explains the impact of each setting with regards to the space used in the cursor cache and the query performance.









CURSOR_SHARING VALUESPACE USED IN SHARED POOLQUERY PERFORMANCE
EXACT (No literal replacement)Worst possible case - each stmt issued has its own parent cursorBest possible case as each stmt has its own plan generated for it based on the value of the literal value present in the stmt
FORCEBest possible case as only one parent and child cursor for each distinct stmtPotentially the worst case as only one plan will be used for each distinct stmt and all occurrences of that stmt will use that plan
SIMILAR without histogram presentBest possible case as only one parent and child cursor for each distinct stmtPotentially the worst case as only one plan will be used for each distinct stmt and all occurrences of that stmt will use that plan
SIMILAR with histogram presentNot quite as much space used as with EXACT but close. Instead of each stmt having its own parent cursor they will have their own child cursor (which uses less space)Best possible case as each stmt has its own plan generated for it based on the value of the literal value present in the stmt



In this case the statement with hundreds of children falls into the last category in the above table, having CURSOR_SHARING set to SIMILAR and a histogram on the columns used in the where clause predicate of the statement. The presence of the histogram tells the optimizer that there is a data skew in that column. The data skew means that there could potentially be multiple execution plans for this statement depending on the literal value used. In order to ensure we don't impact the performance of the application, we will peek at the bind variable values and create a new child cursor for each distinct value. Thus ensuring each bind variable value will get the most optimal execution plan. It's probably easier to understand this issue by looking at an example. Let's assume there is an employees table with a histogram on the job column and CURSOR_SHARING has been set to similar. The following query is issued

select * from employees where job = 'Clerk';

The literal value 'Clerk' will be replaced by a system generated bind variable B1 and a parent cursor will be created as

select * from employees where job = :B1;

The optimizer will peek the bind variable B1 and use the literal value 'Clerk' to determine the execution plan. 'Clerk' is a popular value in the job column and so a full table scan plan is selected and child cursor C1 is created for this plan. The next time the query is executed the where clause predicate is job='VP' so B1 will be set to 'VP', this is not a very popular value in the job column so an index range scan is selected and child cursor C2 is created. The third time the query is executed the where clause predicate is job ='Engineer' so the value for B1 is set to 'Engineer'. Again this is a popular value in the job column and so a full table scan plan is selected and a new child cursor C3 is created. And so on until we have seen all of the distinct values for job column. If B1 is set to a previously seen value, say 'Clerk', then we would reuse child cursor C1.







Value for B1Plan UsedCursor Number
ClerkFull Table ScanC1
VPIndex Range ScanC2
EngineerFull Table ScanC3



As each of these cursors is actually a child cursor and not a new parent cursor you will still be better off than with CURSOR_SHARING set to EXACT as a child cursor takes up less space in the cursor cache. A child cursor doesn't contain all of the information stored in a parent cursor, for example, the SQL text is only stored in the parent cursor and not in each child.

Now that you know the explanation for all of the child cursors you are seeing you need to decide if it is a problem for you and if so which aspect affects you most, space used in the SHARED_POOL or query performance. If your goal is to guarantee the application performance is not affected by setting CURSOR_SHARING to SIMILAR then keep the system settings unchanged. If your goal is to reduce the space used in the shared pool then you can use one of the following solutions with different scopes:

  1. Individual SQL statements - drop the histograms on the columns for each of the affected SQL statements

  2. System-wide - set CURSOR_SHARING to FORCE this will ensure only one child cursor per SQL statement


Both of these solutions require testing to ensure you get the desired effect on your system. Oracle Database 11g provides a much better solution using the Adaptive Cursor Sharing feature. In Oracle Database 11g, all you need to do is set CURSOR_SHARING to FORCE and keep the histograms. With Adaptive Cursor Sharing, the optimizer will create a cursor only when its plan is different from any of the plans used by other child cursors. So in the above example, you will get two child cursors (C1 and C2) instead of 3.

06:30 Why are there more cursors in 11g for my query containing bind variables? (18074 Bytes) » Inside the Oracle Optimizer - Removing the black magic
Oracle introduced a new feature, adaptive cursor sharing, in 11g, to improve the plans that are selected for queries containing bind variables. This feature can result in more cursors for the same query containing bind variables. We'll explain why in this article. Before we get into the details, let's review a little history.

Oracle introduced the bind peeking feature in Oracle 9i. With bind peeking, the Optimizer peeks at the values of user-defined bind variables on the first invocation of a cursor. This allows the optimizer to determine the selectivity of any WHERE clause condition as if literals have been used instead of bind variables, thus improving the quality of the execution plan generated for statements using bind variables.

However, there was a problem with this approach, when the column used in the WHERE clause with the bind contained a data skew. If there is data skew in the column, it is likely that a histogram has been created on this column during statistics gathering. When the optimizer peeks at the value of the user-defined bind variable and chooses a plan, it is not guaranteed that this plan will be good for all possible values for the bind variable. In other words, the plan is optimized for the peeked value of the bind variable, but not for all possible values.

In 11g, the optimizer has been enhanced to allow multiple execution plans to be used for a single statement that uses bind variables. This ensures that the best execution plan will be used depending on the bind value. Let's look at an example to see exactly how this works.

Assume I have simple table emp which has 100,000 rows and has one index called emp_i1 on deptno column.

SQL> desc emp

Name Null? Type
---------------------- -------- ----------------------------------
ENAME VARCHAR2(20)
EMPNO NUMBER
PHONE VARCHAR2(20)
DEPTNO NUMBER

There is a data skew in the deptno column, so when I gathered statistics on the emp table, Oracle automatically created a histogram on the deptno column.

SQL> select table_name, column_name, histogram from user_tab_cols;

TABLE_NAME COLUMN_NAME HISTOGRAM
------------------ ------------------ ---------------
EMP DEPTNO HEIGHT BALANCED
EMP EMPNO NONE
EMP ENAME NONE
EMP PHONE NONE


Now I will execute a simple select on my emp table, which has a single WHERE
clause predicate on the deptno column. The predicate contains a bind variable. We will begin by using the value 9 for this bind variable. The value 9 occurs 10 times in the table, i.e. in 0.0001% of the rows.

SQL> exec :deptno := 9

SQL> select /*ACS_1*/ count(*), max(empno)
2 from emp
3 where deptno = :deptno;


COUNT(*) MAX(EMPNO)
---------- ----------
10 99
Given how selective the value 9 is, we should expect to get an index range scan for this query. Lets check the execution plan.

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------
SQL_ID 272gr4hapc9w1, child number 0
------------------------------------------------------------------------
select /*ACS_1*/ count(*), max(empno) from emp where deptno = :deptno

Plan hash value: 3184478295
------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |Cost (%CPU)|
------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 2 (100)|
| 1 | SORT AGGREGATE | | 1| 16 | |
| 2 | TABLE ACCESS BY INDEX ROWID| EMP | 1| 16 | 2 (0)|
| 3 | INDEX RANGE SCAN | EMP_I1| 1| | 1 (0)|
------------------------------------------------------------------------


So we got the index range scan that we expected. Now let's look at the execution statistics for this statement

SQL> select child_number, executions, buffer_gets,
2 is_bind_sensitive, is_bind_aware
3 from v$sql
4 where sql_text like 'select /*ACS_1%';

CHILD_NUMBER EXECUTIONS BUFFER_GETS IS_BIND_SENSITIVE IS_BIND_AWARE
------------ ---------- ----------- ----------------- -------------
0 1 53 Y N

You can see we have one child cursor that has been executed once and has a small number of buffer gets. We also see that the cursor has been marked bind sensitive. A cursor is marked bind sensitive if the optimizer believes the optimal plan may depend on the value of the bind variable. When a cursor is marked bind sensitive, Oracle monitors the behavior of the cursor using different bind values, to determine if a different plan for different bind values is called for. This cursor was marked bind sensitive because the histogram on the deptno column was used to compute the selectivity of the predicate "where deptno = :deptno". Since the presence of the histogram indicates that the column is skewed, different values of the bind variable may call for different plans.

Now let's change the value of the bind variable to 10, which is the most popular value for the deptno column. It occurs 99900 times in the table, i.e in 99.9% of the rows.

SQL>  exec :deptno := 10

SQL> select /*ACS_1*/ count(*), max(empno)
2 from emp
3 where deptno = :deptno;

COUNT(*) MAX(EMPNO)
---------- ----------
99900 100000

We expect to get the same plan as before for this execution because Oracle initially assumes it can be shared. Let's check:

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------
SQL_ID 272gr4hapc9w1, child number 0
------------------------------------------------------------------------
select /*ACS_1*/ count(*), max(empno) from emp where deptno = :deptno

Plan hash value: 3184478295
------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |Cost (%CPU)|
------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 2 (100)|
| 1 | SORT AGGREGATE | | 1| 16 | |
| 2 | TABLE ACCESS BY INDEX ROWID| EMP | 1| 16 | 2 (0)|
| 3 | INDEX RANGE SCAN | EMP_I1| 1| | 1 (0)|
------------------------------------------------------------------------

The plan is still an index range scan as before, but if we look at the execution statistics, we should see two executions and a big jump in the number of buffer gets from what we saw before.

SQL> select child_number, executions, buffer_gets,
2 is_bind_sensitive, is_bind_aware
3 from v$sql
4 where sql_text like 'select /*ACS_1%';

CHILD_NUMBER EXECUTIONS BUFFER_GETS IS_BIND_SENSITIVE IS_BIND_AWARE
------------ ---------- ----------- ----------------- -------------
0 2 1007 Y N

You should also note that the cursor is still only marked bind sensitive and not bind aware at this point. So let's re-execute the statement using the same popular value, 10.

SQL> exec :deptno := 10

SQL> select /*ACS_1*/ count(*), max(empno)
2 from emp
3 where deptno = :deptno;

COUNT(*) MAX(EMPNO)
---------- -----------
99900 100000

Behind the scenes during the first two executions, Oracle was monitoring the behavior of the queries, and determined that the different bind values caused the data volumes manipulated by the query to be significantly different. Based on this difference, Oracle "adapts" its behavior so that the same plan is not always shared for this query. Hence a new plan is generated based on the current bind value, 10.

Let's check what the new plan is.

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------
SQL_ID 272gr4hapc9w1, child number 1
--------------------------------------------------------------------
select /*ACS_1*/ count(*), max(empno) from emp where deptno = :deptno

Plan hash value: 2083865914
--------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)|
--------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 240 (100)|
| 1 | SORT AGGREGATE | | 1 | 16 | |
|* 2 | TABLE ACCESS FULL | EMP | 95000 | 1484K | 240 (1)|
--------------------------------------------------------------------


Given how unselective the value 10 is in the table, it's not surprising that the new plan is a full table scan. Now if we display the execution statistics we should see an additional child cursor (#1) has been created. Cursor #1 should show a number of buffers gets lower than cursor #0 and it is marked both bind sensitive and bind aware. A bind aware cursor may use different plans for different bind values, depending on how selective the predicates containing the bind variable are.

Looking at the execution statistics:

SQL> select child_number, executions, buffer_gets,
2 is_bind_sensitive, is_bind_aware
3 from v$sql
4 where sql_text like 'select /*ACS_1%';

CHILD_NUMBER EXECUTIONS BUFFER_GETS IS_BIND_SENSITIVE IS_BIND_AWARE
------------ ---------- ----------- ----------------- -------------
0 2 1007 Y N
1 1 821 Y Y

we see that there is a new cursor, which represents the plan which uses a table scan. But if we execute the query again with a more selective bind value, we should use the index plan:

SQL> exec :deptno := 9

SQL> select /*ACS_1*/ count(*), max(empno)
2 from emp
3 where deptno = :deptno;

COUNT(*) MAX(EMPNO)
---------- ----------
10 99

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------
SQL_ID 272gr4hapc9w1, child number 2
------------------------------------------------------------------------
select /*ACS_1*/ count(*), max(empno) from emp where deptno = :deptno

Plan hash value: 3184478295
------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |Cost (%CPU)|
------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 2 (100)|
| 1 | SORT AGGREGATE | | 1| 16 | |
| 2 | TABLE ACCESS BY INDEX ROWID| EMP | 1| 16 | 2 (0)|
| 3 | INDEX RANGE SCAN | EMP_I1| 1| | 1 (0)|
------------------------------------------------------------------------

The proper plan was chosen, based on the selectivity produced by the current bind value.

There is one last interesting thing to note about this. If we look at the execution statistics again, there are three cursors now:


SQL> select child_number, executions, buffer_gets,
2 is_bind_sensitive, is_bind_aware, is_shareable
3 from v$sql
4 where sql_text like 'select /*ACS_1%';

CHILD_NUMBER EXECUTIONS BUFFER_GETS IS_B_SENS IS_B_AWAR IS_SHAR
------------ ---------- ----------- --------- --------- ----------
0 2 957 Y N N
1 1 765 Y Y Y
2 2 6 Y Y Y

The original cursor was discarded when the cursor switched to bind aware mode. This is a one-time overhead. Note that the cursor is marked as not shareable (is_shareable is "N"), which means that this cursor will be among the first to be aged out of the cursor cache, and that it will no longer be used. In other words, it is just waiting to be garbage collected.

There is one other reason that you may notice additional cursors for such a query in 11g. When a new bind value is used, the optimizer tries to find a cursor that it thinks will be a good fit, based on similarity in the bind value's selectivity. If it cannot find such a cursor, it will create a new one (like above, when one (#1) was created for unselective "10" and one (#2) was created for highly-selective "9"). If the plan for the new cursor is the same as one of the existing cursors, the two cursors will be merged, to save space in the cursor cache. This will result in one being left behind that is in a not shareable state. This cursor will be aged out first if there is crowding in the cursor cache, and will not be used for future executions.

Q & A
Instead of answering the questions in your comments one by one, I am going to summarize the questions and provide my answers here.

Q: Is this behavior managed by 11g optimizer automatically and we don't need cursor_sharing anymore?
A: We have not changed the behavior of the cursor_sharing parameter yet, for backwards compatibility purposes. So if you set it to similar, adaptive cursor sharing will only kick in for queries where the literals are replace with binds. We hope that in the future, this feature will persuade people to set cursor_sharing to force.

Q: Would it have any impact like holding library cache latches for longer time to search for appropriate child cursor.
A: Any additional overhead in matching a cursor is always a concern, and we strive to minimize the impact. There is of course some increase in the code path to match a bind-aware cursor, since it requires more intelligent checks. This feature should not, however, impact cursors which are not yet marked bind-aware.

Q: What triggers a cursor to be marked "bind sensitive"?
A: Our goal is to consider many types of predicates where the selectivity can change when the bind value changes.
In this first version of the feature, we only handle equality predicates where a histogram exists on the column and range predicates (with or without histogram). We do not currently consider LIKE predicates, but it is on the top of our list for future work.

Q: Also it sounds like the optimizer is using the number of rows returned to decided that it's time for a new plan...
A: I am not going to go into the details of the "special sauce" for how we decide to mark a cursor bind-aware. The number of rows processed is one input.

Q: Are you planning a hint to mark statements as bind-aware ?
A: Yes, we plan to add this in the future. This will allow users to bypass the startup cost of automatically determining that a query is a good candidate for bind-aware cursor sharing.
06:30 Why are some of the tables in my query missing from the plan? (12762 Bytes) » Inside the Oracle Optimizer - Removing the black magic
We apologize for our brief hiatus from blogging. We've been busy working on improvements to the optimizer.

In 10gR2, we introduced a new transformation, table elimination (alternately called "join elimination"), which removes redundant tables from a query. A table is redundant if its columns are only referenced to in join predicates, and it is guaranteed that those joins neither filter nor expand the resulting rows. There are several cases where Oracle will eliminate a redundant table. We will discuss each case in turn.

Primary Key-Foreign Key Table Elimination

Starting in 10gR2, the optimizer eliminates tables that are redundant due to primary key-foreign key constraints. Consider the following example tables:

create table jobs
(
job_id NUMBER PRIMARY KEY,

job_title VARCHAR2(35) NOT NULL,
min_salary NUMBER,
max_salary NUMBER
);

create table departments
(
department_id NUMBER PRIMARY KEY,
department_name VARCHAR2(50)
);
create table employees
(
employee_id NUMBER PRIMARY KEY,
employee_name VARCHAR2(50),
department_id NUMBER REFERENCES departments(department_id),
job_id NUMBER REFERENCES jobs(job_id)
);

and the query:

select e.employee_name
from employees e, departments d
where e.department_id = d.department_id;

In this query, the join to departments is redundant. The only column referenced in the query appears in the join predicate, and the primary key-foreign key constraint guarantees that there is at most one match in departments for each row in employees. Hence, the query is equivalent to:

select e.employee_name
from employees e
where e.department_id is not null;


The optimizer will generate this plan for the query:

-------------------------------------------
Id Operation Name
-------------------------------------------
0 SELECT STATEMENT
* 1 TABLE ACCESS FULL EMPLOYEES
-------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter("E"."DEPARTMENT_ID" IS NOT NULL)

Note that the IS NOT NULL predicate is not necessary if the column has a NOT NULL constraint on it.

Starting in 11gR1, the optimizer will also eliminate tables that are semi-joined or anti-joined. Consider the following query:

select e.employee_id, e.employee_name
from employees e
where not exists (select 1
from jobs j
where j.job_id = e.job_id);


Since employees.job_id is a foreign key to jobs.job_id, any non-null value in employees.job_id must have a match in jobs. So only employees with null values for employees.job_id will appear in the result. Hence, this query is equivalent to:

select e.employee_id, e.employee_name
from employees e
where job_id is null;

and the optimizer can choose this plan:

-------------------------------------------
Id Operation Name
-------------------------------------------
0 SELECT STATEMENT
* 1 TABLE ACCESS FULL EMPLOYEES
-------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter("E"."JOB_ID" IS NULL)

Suppose employees.job_id has a NOT NULL constraint:

alter table employees modify job_id not null;

In this case, there could not possibly be any rows in EMPLOYEES, and the optimizer could choose this plan:

-------------------------------------------
Id Operation Name
-------------------------------------------
0 SELECT STATEMENT
* 1 FILTER
2 TABLE ACCESS FULL EMPLOYEES
-------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(NULL IS NOT NULL)

The "NULL IS NOT NULL" filter is a false constant predicate, that will prevent the table scan from even taking place.

Also in 11gR1, the optimization became available for ANSI compliant joins. For this query:

select employee_name
from employees e inner join jobs j
on e.job_id = j.job_id;

the optimizer can eliminate JOBS and produce this plan:

-------------------------------------------
Id Operation Name
-------------------------------------------
0 SELECT STATEMENT
1 TABLE ACCESS FULL EMPLOYEES
-------------------------------------------

Outer Join Table Elimination

In 11gR1, a new form of table elimination was introduced for outer joins, which does not require PK-FK constraints. For the example, we require a new table and an addition to EMPLOYEES:

create table projects
(
project_id NUMBER UNIQUE,
deadline DATE,
priority NUMBER
);

alter table employees add project_id number;

Now consider a query that outer joins employees and projects:

select e.employee_name, e.project_id
from employees e, projects p
where e.project_id = p.project_id (+);

The outer join guarantees that every row in employees will appear at least once in the result. The unique constraint on projects.project_id guarantees that every row in employees will match at most one row in projects. Together, these two properties guarantee that every row in employees will appear in the result exactly once. Since no other columns from projects are referenced, projects can be eliminated, and the optimizer can choose this plan:

-------------------------------------------
Id Operation Name
-------------------------------------------
0 SELECT STATEMENT
1 TABLE ACCESS FULL EMPLOYEES
-------------------------------------------

Why Would I Ever Write Such a Query?

All of the example queries in this post are very simple, and one would be unlikely to write a query where the join is so obviously unnecessary. There are many real world scenarios where table elimination may be helpful, including machine-generated queries and elimination of tables in views. For example, a set of tables might be exposed as a view, which contains a join. The join may be necessary to retrieve all of the columns exposed by the view. But some users of the view may only access a subset of the columns, and in this case, the joined table can be eliminated.

For example, consider the view:

create view employee_directory_v as
select e.employee_name, d.department_name, j.job_title
from employees e, departments d, jobs j
where e.department_id = d.department_id
and e.job_id = j.job_id;

This view might be exposed to a simple employee directory application. To lookup employee names by job title, the application issues a query:

select employee_name
from employee_directory_v
where department = 'ACCOUNTING';

Since the job_title column is not referenced, jobs can be eliminated from the query, and the optimizer can choose this plan:

--------------------------------------------
Id Operation Name
--------------------------------------------
0 SELECT STATEMENT
* 1 HASH JOIN
2 TABLE ACCESS FULL EMPLOYEES
* 3 TABLE ACCESS FULL DEPARTMENTS
--------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
1 - access("E"."DEPARTMENT_ID"="D"."DEPARTMENT_ID")
3 - filter("D"."DEPARTMENT_NAME"='ACCOUNTING')

Known Limitations

There are currently a few limitations of table elimination:
  • Multi-column primary key-foreign key constraints are not supported.
  • Referring to the join key elsewhere in the query will prevent table elimination. For an inner join, the join keys on each side of the join are equivalent, but if the query contains other references to the join key from the table that could otherwise be eliminated, this prevents elimination. A workaround is to rewrite the query to refer to the join key from the other table (we realize this is not always possible).
06:30 What's Changed between my New Query Plan and the Old One? (240 Bytes) » Inside the Oracle Optimizer - Removing the black magic
06:30 We have moved! (534 Bytes) » Inside the Oracle Optimizer - Removing the black magic
You might have been wondering why things had gone so quiet on the Optimizer development team's blog Optimizer Magic over the last few months. Well the blog has moved to blogs.oracle.com/optimizer. All of the old articles have moved too and we plan to be a lot more active at our new home, with at least one new post every month.


06:30 Upgrading from Oracle Database 9i to 10g: What to expect from the Optimizer (907 Bytes) » Inside the Oracle Optimizer - Removing the black magic
One of the most daunting activities a DBA can undertake is upgrading the database to a new version. Having to comprehend all of the new features and to deal with potential plan changes can be overwhelming. In order to help DBA's upgrade from Oracle Database 9i to 10g a new whitepaper called "Upgrading from Oracle Database 9i to 10g: What to expect from the Optimizer" has recently been posted on Oracle Technology Network (OTN). This paper aims to explain in detail what to expect from the CBO when you upgrade from Oracle database 9i to 10g and describes what steps you should take before and after the upgrade to minimize any potential SQL regressions. This is a must read for any DBA planning on upgrading from 9i to 10g in the near future!
06:30 Understanding DBMS_STATS.SET_*_PREFS procedures (6097 Bytes) » Inside the Oracle Optimizer - Removing the black magic
In previous Database releases you had to use the DBMS_STATS.SET_PARM procedure to change the default value for the parameters used by the DBMS_STATS.GATHER_*_STATS procedures. The scope of any changes that were made was all subsequent operations. In Oracle Database 11g, the DBMS_STATS.SET_PARM procedure has been deprecated and it has been replaced with a set of procedures that allow you to set a preference for each parameter at a table, schema, database, and Global level. These new procedures are called DBMS_STATS.SET_*_PREFS and offer a much finer granularity of control.

However there has been some confusion around which procedure you should use when and what the hierarchy is among these procedures. In this post we hope to clear up the confusion. Lets start by looking at the list of parameters you can change using the DBMS_STAT.SET_*_PREFS procedures.

  • AUTOSTATS_TARGET (SET_GLOBAL_PREFS only)
  • CASCADE
  • DEGREE
  • ESTIMATE_PERCENT
  • METHOD_OPT
  • NO_INVALIDATE
  • GRANULARITY
  • PUBLISH
  • INCREMENTAL
  • STALE_PERCENT

As mentioned above there are four DBMS_STATS.SET_*_PREFS procedures.

  1. SET_TABLE_PREFS

  2. SET_SCHEMA_PREFS

  3. SET_DATABASE_PREFS

  4. SET_GLOBAL_PREFS


The DBMS_STATS.SET_TABLE_PREFS procedure allows you to change the default values of the parameters used by the DBMS_STATS.GATHER_*_STATS procedures for the specified table only.

The DBMS_STATS.SET_SCHEMA_PREFS procedure allows you to change the default values of the parameters used by the DBMS_STATS.GATHER_*_STATS procedures for all of the existing objects in the specified schema. This procedure actually calls DBMS_STATS.SET_TABLE_PREFS for each of the tables in the specified schema. Since it uses DBMS_STATS.SET_TABLE_PREFS calling this procedure will not affect any new objects created after it has been run. New objects will pick up the GLOBAL_PREF values for all parameters.

The DBMS_STATS.SET_DATABASE_PREFS procedure allows you to change the default values of the parameters used by the DBMS_STATS.GATHER_*_STATS procedures for all of the user defined schemas in the database. This procedure actually calls DBMS_STATS.SET_TABLE_PREFS for each of the tables in each of the user defined schemas. Since it uses DBMS_STATS.SET_TABLE_PREFS this procedure will not affect any new objects created after it has been run. New objects will pick up the GLOBAL_PREF values for all parameters. It is also possible to include the Oracle owned schemas (sys, system, etc) by setting the ADD_SYS parameter to TRUE.

The DBMS_STATS.SET_GLOBAL_PREFS procedure allows you to change the default values of the parameters used by the DBMS_STATS.GATHER_*_STATS procedures for any object in the database that does not have an existing table preference. All parameters default to the global setting unless there is a table preference set or the parameter is explicitly set in the DBMS_STATS.GATHER_*_STATS command. Changes made by this procedure will affect any new objects created after it has been run as new objects will pick up the GLOBAL_PREF values for all parameters.

With GLOBAL_PREFS it is also possible to set a default value for one additional parameter, called AUTOSTAT_TARGET. This additional parameter controls what objects the automatic statistic gathering job (that runs in the nightly maintenance window) will look after. The possible values for this parameter are ALL,ORACLE, and AUTO. ALL means the automatic statistics gathering job will gather statistics on all objects in the database. ORACLE means that the automatic statistics gathering job will only gather statistics for Oracle owned schemas (sys, sytem, etc) Finally AUTO (the default) means Oracle will decide what objects to gather statistics on. Currently AUTO and ALL behave the same.

In summary, DBMS_STATS obeys the following hierarchy for parameter values, parameters values set in the DBMS_STAT.GATHER*_STATS command over rules everything. If the parameter has not been set in the command we check for a table level preference. If there is no table preference set we use the global preference.
06:30 SQL Plan Management (Part 4 of 4): User Interfaces and Other Features (29076 Bytes) » Inside the Oracle Optimizer - Removing the black magic
In the first three parts of this article, we have seen how SQL plan baselines are created, used and evolved. In this final installment, we will show some user interfaces, describe the interaction of SPM with other features and answer some of your questions.

DBMS_SPM package

A new package, DBMS_SPM, allows you to manage plan histories. We have already seen in previous examples how you can use it to create and evolve SQL plan baselines. Other management functions include changing attributes (like enabled status and plan name) of plans or dropping plans. You need the ADMINISTER SQL MANAGEMENT OBJECT privilege to execute this package.

Viewing the plan history

Regardless of how a plan history is created, you can view details about the various plans in the view DBA_SQL_PLAN_BASELINES. At the end of Part 3 of this blog, we saw that the SQL statement had two accepted plans:

SQL> select sql_text, sql_handle, plan_name, enabled, accepted
  2  from dba_sql_plan_baselines;


SQL_TEXT                 SQL_HANDLE               PLAN_NAME                     ENA ACC
------------------------ ------------------------ ----------------------------- --- ---
select p.prod_name, s.am SYS_SQL_4bf04d85fcc170b0 SYS_SQL_PLAN_fcc170b08cbcb825 YES YES
ount_sold, t.calendar_ye
ar
from sales s, products p
, times t
where s.prod_id = p.prod
_id
  and s.time_id = t.time
_id
  and p.prod_id < :pid

select p.prod_name, s.am SYS_SQL_4bf04d85fcc170b0 SYS_SQL_PLAN_fcc170b0a62d0f4d YES YES
ount_sold, t.calendar_ye
ar
from sales s, products p
, times t
where s.prod_id = p.prod
_id
  and s.time_id = t.time
_id
  and p.prod_id < :pid



The SQL handle is a unique identifier for each SQL statement that you can use when managing your plan history using the DBMS_SPM package.

Creating an accepted plan by modifying the SQL text

Some of you may be manually tuning SQL statements by adding hints or otherwise modifying the SQL text. If you enable automatic capture of SQL plans and then execute this statement, you will be creating a SQL plan baseline for this modified statement. What you most likely want, however, is to add this plan to the plan history of the original SQL statement. Here's how you can do this using the above SQL statement as an example.

Let's modify the SQL statement, execute it and look at the plan:


SQL> var pid number
SQL> exec :pid := 100;

PL/SQL procedure successfully completed.

SQL> select /*+ leading(t) */ p.prod_name, s.amount_sold, t.calendar_year
2    from sales s, products p, times t
3    where s.prod_id = p.prod_id
4      and s.time_id = t.time_id
5      and p.prod_id < :pid;

PROD_NAME AMOUNT_SOLD CALENDAR_YEAR
--------- ----------- -------------
...
9 rows selected.

SQL> select * from table(dbms_xplan.display_cursor('b17wnz4y8bqv1', 0, 'basic note'));

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------------
EXPLAINED SQL STATEMENT:
------------------------
select /*+ leading(t) */ p.prod_name, s.amount_sold, t.calendar_year
from sales s, products p, times t where s.prod_id = p.prod_id   and
s.time_id = t.time_id   and p.prod_id < :pid

Plan hash value: 2290436051

---------------------------------------------------------------
| Id  | Operation                            | Name           |
---------------------------------------------------------------
|   0 | SELECT STATEMENT                     |                |
|   1 |  HASH JOIN                           |                |
|   2 |   HASH JOIN                          |                |
|   3 |    TABLE ACCESS FULL                 | TIMES          |
|   4 |    PARTITION RANGE ALL               |                |
|   5 |     TABLE ACCESS BY LOCAL INDEX ROWID| SALES          |
|   6 |      BITMAP CONVERSION TO ROWIDS     |                |
|   7 |       BITMAP INDEX RANGE SCAN        | SALES_PROD_BIX |
|   8 |   TABLE ACCESS BY INDEX ROWID        | PRODUCTS       |
|   9 |    INDEX RANGE SCAN                  | PRODUCTS_PK    |
---------------------------------------------------------------


23 rows selected.


We can now create a new accepted plan for the original SQL statement by associating the modified statement's plan to the original statement's sql handle (obtained from DBA_SQL_PLAN_BASELINES):

SQL> var pls number
SQL> exec :pls := dbms_spm.load_plans_from_cursor_cache( -
>                    sql_id => 'b17wnz4y8bqv1', -
>                    plan_hash_value => 2290436051, -
>                    sql_handle => 'SYS_SQL_4bf04d85fcc170b0');


If the original SQL statement does not already have a plan history (and thus no SQL handle), another version of load_plans_from_cursor_cache allows you to specify the original statement's text.

To confirm that we now have three accepted plans for our SQL statement, let's check in DBA_SQL_PLAN_BASELINES:

SQL> select sql_text, sql_handle, plan_name, enabled, accepted
  2  from dba_sql_plan_baselines;



SQL_TEXT                 SQL_HANDLE               PLAN_NAME                     ENA ACC
------------------------ ------------------------ ----------------------------- --- ---
select p.prod_name, s.am SYS_SQL_4bf04d85fcc170b0 SYS_SQL_PLAN_fcc170b0888547d3 YES YES
ount_sold, t.calendar_ye
ar
from sales s, products p
, times t
where s.prod_id = p.prod
_id
  and s.time_id = t.time
_id
  and p.prod_id < :pid

select p.prod_name, s.am SYS_SQL_4bf04d85fcc170b0 SYS_SQL_PLAN_fcc170b08cbcb825 YES YES
ount_sold, t.calendar_ye
ar
from sales s, products p
, times t
where s.prod_id = p.prod
_id
  and s.time_id = t.time
_id
  and p.prod_id < :pid

select p.prod_name, s.am SYS_SQL_4bf04d85fcc170b0 SYS_SQL_PLAN_fcc170b0a62d0f4d YES YES
ount_sold, t.calendar_ye
ar
from sales s, products p
, times t
where s.prod_id = p.prod
_id
  and s.time_id = t.time
_id
  and p.prod_id < :pid



Displaying plans

When the optimizer uses an accepted plan for a SQL statement, you can see it in the plan table (for explain) or V$SQL_PLAN (for shared cursors). Let's explain the SQL statement above and display its plan:

SQL> explain plan for
  2  select p.prod_name, s.amount_sold, t.calendar_year
  3  from sales s, products p, times t
  4  where s.prod_id = p.prod_id
  5    and s.time_id = t.time_id
  6    and p.prod_id < :pid;

Explained.

SQL> select * from table(dbms_xplan.display('plan_table', null, 'basic note'));

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------------
Plan hash value: 2787970893

----------------------------------------------------------------
| Id  | Operation                             | Name           |
----------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                |
|   1 |  NESTED LOOPS                         |                |
|   2 |   NESTED LOOPS                        |                |
|   3 |    HASH JOIN                          |                |
|   4 |     TABLE ACCESS BY INDEX ROWID       | PRODUCTS       |
|   5 |      INDEX RANGE SCAN                 | PRODUCTS_PK    |
|   6 |     PARTITION RANGE ALL               |                |
|   7 |      TABLE ACCESS BY LOCAL INDEX ROWID| SALES          |
|   8 |       BITMAP CONVERSION TO ROWIDS     |                |
|   9 |        BITMAP INDEX RANGE SCAN        | SALES_PROD_BIX |
|  10 |    INDEX UNIQUE SCAN                  | TIME_PK        |
|  11 |   TABLE ACCESS BY INDEX ROWID         | TIMES          |
----------------------------------------------------------------

Note
-----
- SQL plan baseline "SYS_SQL_PLAN_fcc170b0a62d0f4d" used for this statement

22 rows selected.



The note at the bottom tells you that the optimizer used an accepted plan.

A plan history might have multiple plans. You can see one of the accepted plans if the optimizer selects it for execution. But what if you want to display some or all of the other plans? You can do this using the display_sql_plan_baseline function in the DBMS_XPLAN package. Using the above example, here's how you can display the plan for all plans in the plan history.

SQL> select *
  2  from table(dbms_xplan.display_sql_plan_baseline(
  3               sql_handle => 'SYS_SQL_4bf04d85fcc170b0', format => 'basic'));



PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
SQL handle: SYS_SQL_4bf04d85fcc170b0
SQL text: select p.prod_name, s.amount_sold, t.calendar_year from sales s,
          products p, times t where s.prod_id = p.prod_id   and s.time_id =
          t.time_id   and p.prod_id < :pid
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Plan name: SYS_SQL_PLAN_fcc170b0888547d3
Enabled: YES     Fixed: NO      Accepted: YES     Origin: MANUAL-LOAD
--------------------------------------------------------------------------------

Plan hash value: 2290436051

---------------------------------------------------------------
| Id  | Operation                            | Name           |
---------------------------------------------------------------
|   0 | SELECT STATEMENT                     |                |
|   1 |  HASH JOIN                           |                |
|   2 |   HASH JOIN                          |                |
|   3 |    TABLE ACCESS FULL                 | TIMES          |
|   4 |    PARTITION RANGE ALL               |                |
|   5 |     TABLE ACCESS BY LOCAL INDEX ROWID| SALES          |
|   6 |      BITMAP CONVERSION TO ROWIDS     |                |
|   7 |       BITMAP INDEX RANGE SCAN        | SALES_PROD_BIX |
|   8 |   TABLE ACCESS BY INDEX ROWID        | PRODUCTS       |
|   9 |    INDEX RANGE SCAN                  | PRODUCTS_PK    |
---------------------------------------------------------------

--------------------------------------------------------------------------------
Plan name: SYS_SQL_PLAN_fcc170b08cbcb825
Enabled: YES     Fixed: NO      Accepted: YES     Origin: AUTO-CAPTURE
--------------------------------------------------------------------------------

Plan hash value: 2361178149

------------------------------------------
| Id  | Operation             | Name     |
------------------------------------------
|   0 | SELECT STATEMENT      |          |
|   1 |  HASH JOIN            |          |
|   2 |   HASH JOIN           |          |
|   3 |    PARTITION RANGE ALL|          |
|   4 |     TABLE ACCESS FULL | SALES    |
|   5 |    TABLE ACCESS FULL  | TIMES    |
|   6 |   TABLE ACCESS FULL   | PRODUCTS |
------------------------------------------

--------------------------------------------------------------------------------
Plan name: SYS_SQL_PLAN_fcc170b0a62d0f4d
Enabled: YES     Fixed: NO      Accepted: YES     Origin: AUTO-CAPTURE
--------------------------------------------------------------------------------

Plan hash value: 2787970893

----------------------------------------------------------------
| Id  | Operation                             | Name           |
----------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                |
|   1 |  NESTED LOOPS                         |                |
|   2 |   NESTED LOOPS                        |                |
|   3 |    HASH JOIN                          |                |
|   4 |     TABLE ACCESS BY INDEX ROWID       | PRODUCTS       |
|   5 |      INDEX RANGE SCAN                 | PRODUCTS_PK    |
|   6 |     PARTITION RANGE ALL               |                |
|   7 |      TABLE ACCESS BY LOCAL INDEX ROWID| SALES          |
|   8 |       BITMAP CONVERSION TO ROWIDS     |                |
|   9 |        BITMAP INDEX RANGE SCAN        | SALES_PROD_BIX |
|  10 |    INDEX UNIQUE SCAN                  | TIME_PK        |
|  11 |   TABLE ACCESS BY INDEX ROWID         | TIMES          |
----------------------------------------------------------------

72 rows selected.


Parameters

Two parameters allow you to control SPM. The first, optimizer_capture_sql_plan_baselines, which is FALSE by default, allows you to automatically capture plans. SPM will start managing every repeatable SQL statement that is executed and will create a plan history for it. The first plan that is captured will beautomatically accepted. Subsequent plans for these statements will not be accepted until they are evolved.

The second parameter, optimizer_use_sql_plan_baselines, is TRUE by default. It allows the SPM aware optimizer to use the SQL plan baseline if available when compiling a SQL statement. If you set this parameter to FALSE, the SPM aware optimizer will be disabled and you will get the regular cost-based optimizer which will select the best plan based on estimated cost.

SPM and SQL profiles

A SQL statement can have both a SQL profile and a SQL plan baseline. Such a case was described in Part 3 where we evolved a SQL plan baseline by accepting a SQL profile. In this case, the SPM aware optimizer will use both the SQL profile and the SQL plan baseline. The SQL profile contains additional information that helps the optimizer to accurately cost each accepted plan and select the best one. The SPM aware optimizer may choose a different accepted plan when a SQL profile is present than when it is not.

SPM and stored outlines

It is possible for a SQL statement to have a stored outline as well as a SQL plan baseline. If a stored outline exists for a SQL statement and is enabled for use, then the optimizer will use it, ignoring the SQL plan baseline. In other words, the stored outline trumps a SQL plan baseline. If you are using stored outlines, you can test SPM by creating SQL plan baselines and disabling the stored outlines. If you are satisfied with SPM, you can either drop the stored outlines or leave them disabled. If SPM doesn't work for you (and we would love to know why), you can re-enable the stored outlines.

If you are using stored outlines, be aware of their limitations:


  • You can only have one stored outline at a time for a given SQL statement. This may be fine in some cases, but a single plan is not necessarily the best when the statement is executed under varying conditions (e.g., bind values).
  • The second limitation is related to the first. Stored outlines do not allow for evolution. That is, even if a better plan exists, the stored outline will continue to be used, potentially degrading your system's performance. To get the better plan, you have to manually drop the current stored outline and generate a new one.
  • If an access path (e.g., an index) used in a stored outline is dropped or otherwise becomes unusable, the partial stored outline will continue to be used with the potential of a much worse plan.


One question that readers have is what we plan to do with the stored outlines feature. Here is the official word in Chapter 20 of Oracle's Performance Tuning Guide:

Stored outlines will be desupported in a future release in favor of SQL plan management. In Oracle Database 11g Release 1 (11.1), stored outlines continue to function as in past releases. However, Oracle strongly recommends that you use SQL plan management for new applications. SQL plan management creates SQL plan baselines, which offer superior SQL performance and stability compared with stored outlines.

If you have existing stored outlines, consider migrating them to SQL plan baselines by using the LOAD_PLANS_FROM_CURSOR_CACHE or LOAD_PLANS_FROM_SQLSET procedure of the DBMS_SPM package. When the migration is complete, you should disable or remove the stored outlines.


SPM and adaptive cursor sharing

Adaptive cursor sharing (ACS) may generate multiple cursors for a given bind sensitive SQL statement if it is determined that a single plan is not optimal under all conditions. Each cursor is generated by forcing a hard parse of the statement. The optimizer will normally select the plan with the best cost upon each hard parse.

When you have a SQL plan baseline for a statement, the SPM aware optimizer will select the best accepted plan as the optimal plan. This also applies for the hard parse of a bind sensitive statement. There may be multiple accepted plans, each of which is optimal for different bind sets. With SPM and ACS enabled, the SPM aware optimizer will select the best plan for the current bind set.

Thus, if a hard parse occurs, the normal SPM plan selection algorithm is used regardless of whether a statement is bind sensitive.

Enterprise Manager

You can view SQL plan baselines and configure and manage most SPM tasks through the Enterprise Manager. The screenshots below show two of these tasks.

Setting init.ora parameters for SPM




Loading SQL plan baselines from cursor cache




Further Reading

More details about SPM are available in the Oracle documentation, especially Chapter 15 of the Performance Tuning Guide. There is also a whitepaper, and a paper published in the VLDB 2008 conference. The VLDB paper also has experimental results that show how SPM prevents performance regressions while simultaneously allowing better plans to be used.
06:30 SQL Plan Management (Part 3 of 4): Evolving SQL Plan Baselines (20640 Bytes) » Inside the Oracle Optimizer - Removing the black magic
In the example in Part 2, we saw that the optimizer used an accepted plan instead of a brand new plan. The statement has two plans in its plan history, but only one is accepted and thus in the SQL plan baseline:

SQL> select sql_text, plan_name, enabled, accepted from dba_sql_plan_baselines;
SQL_TEXT                                 PLAN_NAME                      ENA ACC
---------------------------------------- ------------------------------ --- ---
select p.prod_name, s.amount_sold, t.cal SYS_SQL_PLAN_fcc170b08cbcb825  YES NO
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
  and s.time_id = t.time_id
  and p.prod_id < :pid

select p.prod_name, s.amount_sold, t.cal SYS_SQL_PLAN_fcc170b0a62d0f4d  YES YES
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
  and s.time_id = t.time_id
  and p.prod_id < :pid



Non-accepted plans can be verified by executing the evolve_sql_plan_baseline function. This function will execute the non-accepted plan and compare its performance to the best accepted plan. The execution is performed using the conditions (e.g., bind values, parameters, etc.) in effect at the time the non-accepted plan was added to the plan history. If the non-accepted plan's performance is better, the function will make it accepted, thus adding it to the SQL plan baseline. Let's see what happens when we execute this function:

SQL> var report clob;


SQL> exec :report := dbms_spm.evolve_sql_plan_baseline();

PL/SQL procedure successfully completed.

SQL> print :report

REPORT
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
                        Evolve SQL Plan Baseline Report
-------------------------------------------------------------------------------

Inputs:
-------
  SQL_HANDLE =
  PLAN_NAME  =
  TIME_LIMIT = DBMS_SPM.AUTO_LIMIT
  VERIFY     = YES
  COMMIT     = YES

Plan: SYS_SQL_PLAN_fcc170b08cbcb825
-----------------------------------
  Plan was verified: Time used .1 seconds.
  Passed performance criterion: Compound improvement ratio >= 10.13
  Plan was changed to an accepted plan.

                      Baseline Plan      Test Plan     Improv. Ratio
                      -------------      ---------     -------------
  Execution Status:        COMPLETE       COMPLETE
  Rows Processed:               960            960
  Elapsed Time(ms):              19             15              1.27
  CPU Time(ms):                  18             15               1.2
  Buffer Gets:                 1188            116             10.24
  Disk Reads:                     0              0
  Direct Writes:                  0              0
  Fetches:                        0              0
  Executions:                     1              1

-------------------------------------------------------------------------------
                                 Report Summary
-------------------------------------------------------------------------------
Number of SQL plan baselines verified: 1.
Number of SQL plan baselines evolved: 1.



The plan verification report shows that the new plan's performance was better and so it was made accepted and became part of the SQL plan baseline. We can confirm it by looking in the dba_sql_plan_baselines view:

SQL> select sql_text, plan_name, enabled, accepted from dba_sql_plan_baselines;


SQL_TEXT                                 PLAN_NAME                      ENA ACC
---------------------------------------- ------------------------------ --- ---
select p.prod_name, s.amount_sold, t.cal SYS_SQL_PLAN_fcc170b08cbcb825  YES YES
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
  and s.time_id = t.time_id
  and p.prod_id < :pid

select p.prod_name, s.amount_sold, t.cal SYS_SQL_PLAN_fcc170b0a62d0f4d  YES YES
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
  and s.time_id = t.time_id
  and p.prod_id < :pid



The SQL plan baseline now has two accepted plans: SYS_SQL_PLAN_fcc170b08cbcb825 is now accepted.

You can either execute the evolve_sql_plan_baseline() function manually or schedule it to run automatically in a maintenance window.

Another way of evolving a SQL plan baseline is to use the SQL Tuning Advisor. Instead of executing evolve_sql_plan_baseline, suppose we start from the original state where we have one accepted and one non-accepted plan:

SQL> select sql_text, plan_name, enabled, accepted from dba_sql_plan_baselines;


SQL_TEXT                                 PLAN_NAME                      ENA ACC
---------------------------------------- ------------------------------ --- ---
select p.prod_name, s.amount_sold, t.cal SYS_SQL_PLAN_fcc170b08cbcb825  YES NO
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
  and s.time_id = t.time_id
  and p.prod_id < :pid

select p.prod_name, s.amount_sold, t.cal SYS_SQL_PLAN_fcc170b0a62d0f4d  YES YES
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
  and s.time_id = t.time_id
  and p.prod_id < :pid



You can execute the SQL Tuning Advisor on the cursor in the cursor cache:

SQL> var tname varchar2(30);


SQL> exec :tname := dbms_sqltune.create_tuning_task(sql_id => 'bfbr3zrg9d5cc');

PL/SQL procedure successfully completed.

SQL> exec dbms_sqltune.execute_tuning_task(task_name => :tname);

PL/SQL procedure successfully completed.

SQL> select dbms_sqltune.report_tuning_task(:tname, 'TEXT', 'BASIC') FROM dual;

DBMS_SQLTUNE.REPORT_TUNING_TASK(:TNAME,'TEXT','BASIC')
-------------------------------------------------------------------------------
GENERAL INFORMATION SECTION
-------------------------------------------------------------------------------
Tuning Task Name   : TASK_505
Tuning Task Owner  : SH
Workload Type      : Single SQL Statement
Scope              : COMPREHENSIVE
Time Limit(seconds): 1800
Completion Status  : COMPLETED
Started at         : 11/11/2008 16:43:12
Completed at       : 11/11/2008 16:43:13

-------------------------------------------------------------------------------
Schema Name: SH
SQL ID     : bfbr3zrg9d5cc
SQL Text   : select p.prod_name, s.amount_sold, t.calendar_year
             from sales s, products p, times t
             where s.prod_id = p.prod_id
               and s.time_id = t.time_id
               and p.prod_id < :pid

-------------------------------------------------------------------------------
FINDINGS SECTION (1 finding)
-------------------------------------------------------------------------------
1- A potentially better execution plan was found for this statement.

-------------------------------------------------------------------------------
EXPLAIN PLANS SECTION
-------------------------------------------------------------------------------

1- Original With Adjusted Cost
------------------------------
Plan hash value: 2787970893


----------------------------------------------------------------
| Id  | Operation                             | Name           |
----------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                |
|   1 |  NESTED LOOPS                         |                |
|   2 |   NESTED LOOPS                        |                |
|   3 |    HASH JOIN                          |                |
|   4 |     TABLE ACCESS BY INDEX ROWID       | PRODUCTS       |
|   5 |      INDEX RANGE SCAN                 | PRODUCTS_PK    |
|   6 |     PARTITION RANGE ALL               |                |
|   7 |      TABLE ACCESS BY LOCAL INDEX ROWID| SALES          |
|   8 |       BITMAP CONVERSION TO ROWIDS     |                |
|   9 |        BITMAP INDEX RANGE SCAN        | SALES_PROD_BIX |
|  10 |    INDEX UNIQUE SCAN                  | TIME_PK        |
|  11 |   TABLE ACCESS BY INDEX ROWID         | TIMES          |
----------------------------------------------------------------

2- Original With Adjusted Cost
------------------------------
Plan hash value: 2787970893


----------------------------------------------------------------
| Id  | Operation                             | Name           |
----------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                |
|   1 |  NESTED LOOPS                         |                |
|   2 |   NESTED LOOPS                        |                |
|   3 |    HASH JOIN                          |                |
|   4 |     TABLE ACCESS BY INDEX ROWID       | PRODUCTS       |
|   5 |      INDEX RANGE SCAN                 | PRODUCTS_PK    |
|   6 |     PARTITION RANGE ALL               |                |
|   7 |      TABLE ACCESS BY LOCAL INDEX ROWID| SALES          |
|   8 |       BITMAP CONVERSION TO ROWIDS     |                |
|   9 |        BITMAP INDEX RANGE SCAN        | SALES_PROD_BIX |
|  10 |    INDEX UNIQUE SCAN                  | TIME_PK        |
|  11 |   TABLE ACCESS BY INDEX ROWID         | TIMES          |
----------------------------------------------------------------

3- Using SQL Profile
--------------------
Plan hash value: 2361178149


------------------------------------------
| Id  | Operation             | Name     |
------------------------------------------
|   0 | SELECT STATEMENT      |          |
|   1 |  HASH JOIN            |          |
|   2 |   HASH JOIN           |          |
|   3 |    PARTITION RANGE ALL|          |
|   4 |     TABLE ACCESS FULL | SALES    |
|   5 |    TABLE ACCESS FULL  | TIMES    |
|   6 |   TABLE ACCESS FULL   | PRODUCTS |
------------------------------------------

-------------------------------------------------------------------------------


SQL> exec dbms_sqltune.accept_sql_profile(task_name => :tname);

PL/SQL procedure successfully completed.

SQL> select sql_text, plan_name, enabled, accepted from dba_sql_plan_baselines;

SQL_TEXT                                 PLAN_NAME                      ENA ACC
---------------------------------------- ------------------------------ --- ---
select p.prod_name, s.amount_sold, t.cal SYS_SQL_PLAN_fcc170b08cbcb825  YES YES
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
  and s.time_id = t.time_id
  and p.prod_id < :pid

select p.prod_name, s.amount_sold, t.cal SYS_SQL_PLAN_fcc170b0a62d0f4d  YES YES
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
  and s.time_id = t.time_id
  and p.prod_id < :pid

SQL> select sql_text, type, status from dba_sql_profiles;

SQL_TEXT                                 TYPE    STATUS
---------------------------------------- ------- --------
select p.prod_name, s.amount_sold, t.cal MANUAL  ENABLED
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
  and s.time_id = t.time_id
  and p.prod_id < :pid



What we see here is that SQL Tuning Advisor found a tuned plan (that coincidentally happened to be the non-accepted plan in our plan history). When we accepted the recommended SQL profile, the SQL Tuning Advisor created a SQL profile and also changed the non-accepted plan to accepted status, thus evolving the SQL plan baseline to two plans.

Note that the SQL Tuning Advisor may also find a completely new tuned plan, one that is not in the plan history. If you then accept the recommended SQL profile, the SQL Tuning Advisor will create a SQL profile and also add the tuned plan to the SQL plan baseline.

Thus, you can evolve a SQL plan baseline either by executing the evolve_sql_plan_baseline function or by using the SQL Tuning Advisor. New and provably better plans will be added by either of these methods to the SQL plan baseline.

06:30 SQL Plan Management (Part 2 of 4): SPM Aware Optimizer (15776 Bytes) » Inside the Oracle Optimizer - Removing the black magic
(Keep sending your feedback and questions. We'll address them in Part 4.)

In Part 1, we saw how you can create SQL plan baselines. After you create a SQL plan baseline for a statement, subsequent executions of that statement will use the SQL plan baseline. From all the plans in the SQL plan baseline, the optimizer will select the one with the best cost in the current environment (including bind values, current statistics, parameters, etc.). The optimizer will also generate the best-cost plan that it would otherwise have used without a SQL plan baseline. However, this best-cost plan will not be used but instead added to the statement's plan history for later verification. In other words, the optimizer will use a known plan from the SQL plan baseline instead of a new and hitherto unknown plan. This guarantees no performance regression.

Let's see this plan selection process in action. First, we create a SQL plan baseline by enabling automatic plan capture and executing the query twice:

SQL> alter session set optimizer_capture_sql_plan_baselines = true;

Session altered.

SQL> var pid number
SQL> exec :pid := 100;

PL/SQL procedure successfully completed.

SQL> select p.prod_name, s.amount_sold, t.calendar_year
2    from sales s, products p, times t
3    where s.prod_id = p.prod_id
4      and s.time_id = t.time_id
5      and p.prod_id < :pid;

PROD_NAME AMOUNT_SOLD CALENDAR_YEAR
--------- ----------- -------------
...
9 rows selected.

SQL> select * from table(dbms_xplan.display_cursor('bfbr3zrg9d5cc', 0, 'basic note'));

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------------
EXPLAINED SQL STATEMENT:
------------------------
select p.prod_name, s.amount_sold, t.calendar_year from sales s,
products p, times t where s.prod_id = p.prod_id and s.time_id =
t.time_id and p.prod_id < :pid

Plan hash value: 2787970893

----------------------------------------------------------------
| Id  | Operation                             | Name           |
----------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                |
|   1 |  NESTED LOOPS                         |                |
|   2 |   NESTED LOOPS                        |                |
|   3 |    HASH JOIN                          |                |
|   4 |     TABLE ACCESS BY INDEX ROWID       | PRODUCTS       |
|   5 |      INDEX RANGE SCAN                 | PRODUCTS_PK    |
|   6 |     PARTITION RANGE ALL               |                |
|   7 |      TABLE ACCESS BY LOCAL INDEX ROWID| SALES          |
|   8 |       BITMAP CONVERSION TO ROWIDS     |                |
|   9 |        BITMAP INDEX RANGE SCAN        | SALES_PROD_BIX |
|  10 |    INDEX UNIQUE SCAN                  | TIME_PK        |
|  11 |   TABLE ACCESS BY INDEX ROWID         | TIMES          |
----------------------------------------------------------------

25 rows selected.

SQL> select p.prod_name, s.amount_sold, t.calendar_year
2    from sales s, products p, times t
3    where s.prod_id = p.prod_id
4      and s.time_id = t.time_id
5      and p.prod_id < :pid;

PROD_NAME AMOUNT_SOLD CALENDAR_YEAR
--------- ----------- -------------
...
9 rows selected.

SQL> alter session set optimizer_capture_sql_plan_baselines = false;

Session altered.

SQL> select sql_text, plan_name, enabled, accepted from dba_sql_plan_baselines;

SQL_TEXT                                 PLAN_NAME                      ENA ACC
---------------------------------------- ------------------------------ --- ---
select p.prod_name, s.amount_sold, t.cal SYS_SQL_PLAN_fcc170b0a62d0f4d  YES YES
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
and s.time_id = t.time_id
and p.prod_id < :pid



We can see that a SQL plan baseline was created for the statement. Suppose the statement is hard parsed again (we do it here by flushing the shared pool). Let's turn off SQL plan management and execute the query with a different bind value:


SQL> exec :pid := 100000;

PL/SQL procedure successfully completed.

SQL> alter system flush shared_pool;

System altered.

SQL> alter session set optimizer_use_sql_plan_baselines = false;

Session altered.

SQL> select p.prod_name, s.amount_sold, t.calendar_year
2    from sales s, products p, times t
3    where s.prod_id = p.prod_id
4      and s.time_id = t.time_id
5      and p.prod_id < :pid;

PROD_NAME AMOUNT_SOLD CALENDAR_YEAR
--------- ----------- -------------
...
960 rows selected.

SQL> select * from table(dbms_xplan.display_cursor('bfbr3zrg9d5cc', 0, 'basic note'));

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------------
EXPLAINED SQL STATEMENT:
------------------------
select p.prod_name, s.amount_sold, t.calendar_year from sales s,
products p, times t where s.prod_id = p.prod_id and s.time_id = t.time_id and
p.prod_id < :pid

Plan hash value: 2361178149

------------------------------------------
| Id  | Operation             | Name     |
------------------------------------------
|   0 | SELECT STATEMENT      |          |
|   1 |  HASH JOIN            |          |
|   2 |   HASH JOIN           |          |
|   3 |    PARTITION RANGE ALL|          |
|   4 |     TABLE ACCESS FULL | SALES    |
|   5 |    TABLE ACCESS FULL  | TIMES    |
|   6 |   TABLE ACCESS FULL   | PRODUCTS |
------------------------------------------

20 rows selected.



We can see that the optimizer selected a different plan because the new bind value makes the predicate less selective. Let's turn SQL plan management back on and re-execute the query with the same bind value:

SQL> alter session set optimizer_use_sql_plan_baselines = true;

Session altered.

SQL> select p.prod_name, s.amount_sold, t.calendar_year
2    from sales s, products p, times t
3    where s.prod_id = p.prod_id
4      and s.time_id = t.time_id
5      and p.prod_id < :pid;

PROD_NAME AMOUNT_SOLD CALENDAR_YEAR
--------- ----------- -------------
...
960 rows selected.

SQL> select * from table(dbms_xplan.display_cursor('bfbr3zrg9d5cc', 0, 'basic note'));

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------------
EXPLAINED SQL STATEMENT:
------------------------
select p.prod_name, s.amount_sold, t.calendar_year from sales s,
products p, times t where s.prod_id = p.prod_id and s.time_id =
t.time_id and p.prod_id < :pid

Plan hash value: 2787970893

----------------------------------------------------------------
| Id  | Operation                             | Name           |
----------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                |
|   1 |  NESTED LOOPS                         |                |
|   2 |   NESTED LOOPS                        |                |
|   3 |    HASH JOIN                          |                |
|   4 |     TABLE ACCESS BY INDEX ROWID       | PRODUCTS       |
|   5 |      INDEX RANGE SCAN                 | PRODUCTS_PK    |
|   6 |     PARTITION RANGE ALL               |                |
|   7 |      TABLE ACCESS BY LOCAL INDEX ROWID| SALES          |
|   8 |       BITMAP CONVERSION TO ROWIDS     |                |
|   9 |        BITMAP INDEX RANGE SCAN        | SALES_PROD_BIX |
|  10 |    INDEX UNIQUE SCAN                  | TIME_PK        |
|  11 |   TABLE ACCESS BY INDEX ROWID         | TIMES          |
----------------------------------------------------------------

Note
-----
- SQL plan baseline SYS_SQL_PLAN_fcc170b0a62d0f4d used for this statement

29 rows selected.



The note at the bottom tells you that the optimizer is using the SQL plan baseline. In other words, we can see that the optimizer used an accepted plan in the SQL plan baseline in favor of a new plan. In fact, we can also check that the optimizer inserted the new plan into the statement's plan history:

SQL> select sql_text, plan_name, enabled, accepted from dba_sql_plan_baselines;


SQL_TEXT                                 PLAN_NAME                      ENA ACC
---------------------------------------- ------------------------------ --- ---
select p.prod_name, s.amount_sold, t.cal SYS_SQL_PLAN_fcc170b08cbcb825  YES NO
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
  and s.time_id = t.time_id
  and p.prod_id < :pid

select p.prod_name, s.amount_sold, t.cal SYS_SQL_PLAN_fcc170b0a62d0f4d  YES YES
endar_year
from sales s, products p, times t
where s.prod_id = p.prod_id
  and s.time_id = t.time_id
  and p.prod_id < :pid



The 'NO' value for the accepted column implies that the new plan is in the plan history but is not available for use until it is verified to be a good plan. The optimizer will continue to use an accepted plan until new plans are verified and added to the SQL plan baseline. If there is more than one plan in the SQL plan baseline, the optimizer will use the one with the best cost under the then-current conditions (statistics, bind values, parameter settings and so on).

When you create a SQL plan baseline for a SQL statement, the SPM aware optimizer thus guarantees that no new plans will be used other than the ones in the SQL plan baseline. This prevents unexpected plan changes that sometimes lead to performance regressions.

Preventing new plans from being used is fine, but what if the new plans are in fact better? In Part 3, we will describe how new and improved plans are added to a SQL plan baseline.

06:30 Plan regressions got you down? SQL Plan Management to the rescue! (12661 Bytes) » Inside the Oracle Optimizer - Removing the black magic
Part 1 of 4: Creating SQL plan baselines

Do you ever experience performance regressions because an execution plan has changed for the worse? If you have, then we have an elegant solution for you in 11g called SQL Plan Management (SPM). The next four posts on our blog will cover SPM in detail. Let's begin by reviewing the primary causes for plan changes.

Execution plan changes occur due to various system changes. For example, you might have (manually or automatically) updated statistics for some objects, or changed a few optimizer-related parameters. A more dramatic change is a database upgrade (say from 10gR2 to 11g). All of these changes have the potential to cause new execution plans to be generated for many of your SQL statements. Most new plans are obviously improvements because they are tailored to the new system environment, but some might be worse leading to performance regressions. It is the latter that cause sleepless nights for many DBAs.

DBAs have several options for addressing these regressions. However, what most DBAs want is simple: plans should only change when they will result in performance gains. In other words, the optimizer should not pick bad plans, period.

This first post in our series, describes the concepts of SQL Plan Management and how to create SQL plan baselines. The second part will describe how and when these SQL plan baselines are used. The third part will discuss evolution, the process of adding new and improved plans to SQL plan baselines. Finally, the fourth part will describe user interfaces and interactions with other Oracle objects (like stored outlines).


Introduction

SQL Plan Management (SPM) allows database users to maintain stable yet optimal performance for a set of SQL statements. SPM incorporates the positive attributes of plan adaptability and plan stability, while simultaneously avoiding their shortcomings. It has two main objectives:
  1. prevent performance regressions in the face of database system changes
  2. offer performance improvements by gracefully adapting to database system changes
A managed SQL statement is one for which SPM has been enabled. SPM can be configured to work automatically or it can be manually controlled either wholly or partially (described later). SPM helps prevent performance regressions by enabling the detection of plan changes for managed SQL statements. For this purpose, SPM maintains, on disk, a plan history consisting of different execution plans generated for each managed SQL statement. An enhanced version of the Oracle optimizer, called SPM aware optimizer, accesses, uses, and manages this information which is stored in a repository called the SQL Management Base (SMB).

The plan history enables the SPM aware optimizer to determine whether the best-cost plan it has produced using the cost-based method is a brand new plan or not. A brand new plan represents a plan change that has potential to cause performance regression. For this reason, the SPM aware optimizer does not choose a brand new best-cost plan. Instead, it chooses from a set of accepted plans. An accepted plan is one that has been either verified to not cause performance regression or designated to have good performance. A set of accepted plans is called a SQL plan baseline, which represents a subset of the plan history.

A brand new plan is added to the plan history as a non-accepted plan. Later, an SPM utility verifies its performance, and keeps it as a non-accepted plan if it will cause a performance regression, or changes it to an accepted plan if it will provide a performance improvement. The plan performance verification process ensures both plan stability and plan adaptability.

The figure below shows the SMB containing the plan history for three SQL statements. Each plan history contains some accepted plans (the SQL plan baseline) and some non-accepted plans.




(Click on the image for a larger view.)

You can create a SQL plan baseline in several ways: using a SQL Tuning Set (STS); from the cursor cache; exporting from one database and importing into another; and automatically for every statement. Let's look at each in turn. The examples in this blog entry use the Oracle Database Sample Schemas so you can try them yourself.


Creating SQL plan baselines from STS

If you are upgrading from 10gR2 or have an 11g test system, you might already have an STS containing some or all of your SQL statements. This STS might contain plans that perform satisfactorily. Let's call this STS MY_STS. You can create a SQL plan baseline from this STS as follows:

SQL> variable pls number;
SQL> exec :pls := dbms_spm.load_plans_from_sqlset(sqlset_name => 'MY_STS', -
>                   basic_filter => 'sql_text like ''select%p.prod_name%''');


This will create SQL plan baselines for all statements that match the specified filter.

Creating SQL plan baselines from cursor cache

You can automatically create SQL plan baselines for any cursor that is currently in the cache as follows:

SQL> exec :pls := dbms_spm.load_plans_from_cursor_cache( -
>                   attribute_name => 'SQL_TEXT', -
>                   attribute_value => 'select%p.prod_name%');


This will create SQL plan baselines for all statements whose text matches the specified string. Several overloaded variations of this function allow you to filter on other cursor attributes.

Creating SQL plan baselines using a staging table

If you already have SQL plan baselines (say on an 11g test system), you can export them to another system (a production system for instance).

First, on the test system, create a staging table and pack the SQL plan baselines you want to export:

SQL> exec dbms_spm.create_stgtab_baseline(table_name => 'MY_STGTAB', -
>           table_owner => 'SH');

PL/SQL procedure successfully completed.

SQL> exec :pls := dbms_spm.pack_stgtab_baseline( -
>                   table_name => 'MY_STGTAB', -
>                   table_owner => 'SH', -
>                   sql_text => 'select%p.prod_name%');


This will pack all SQL plan baselines for statements that match the specified filter. The staging table, MY_STGTAB, is a regular table that you should export to the production system using Datapump Export.

On the production system, you can now unpack the staging table to create the SQL plan baselines:

SQL> exec :pls := dbms_spm.unpack_stgtab_baseline( -
>                   table_name => 'MY_STGTAB', -
>                   table_owner => 'SH', -
>                   sql_text => 'select%p.prod_name%');


This will unpack the staging table and create SQL plan baselines. Note that the filter for unpacking the staging table is optional and may be different than the one used during packing. This means that you can pack several SQL plan baselines into a staging table and selectively unpack only a subset of them on the target system.

Creating SQL plan baselines automatically

You can create SQL plan baselines for all repeatable statements automatically by setting the parameter optimizer_capture_sql_plan_baselines to TRUE (default is FALSE). The first plan captured for any statement is automatically accepted and becomes part of the SQL plan baseline, so enable this parameter only when you are sure that the default plans are performing well.

You can use the automatic plan capture mode when you have upgraded from a previous database version. Set optimizer_features_enable to the earlier version and execute your workload. Every repeatable statement will have its plan captured thus creating SQL plan baselines. You can reset optimizer_features_enable to its default value after you are sure that all statements in your workload have had a chance to execute.

Note that this automatic plan capture occurs only for repeatable statements, that is, statements that are executed at least twice. Statements that are only executed once will not benefit from SQL plan baselines since accepted plans are only used in subsequent hard parses.

The following example shows a plan being captured automatically when the same statement is executed twice:

SQL> alter session set optimizer_capture_sql_plan_baselines = true;

Session altered.

SQL> var pid number
SQL> exec :pid := 100;

PL/SQL procedure successfully completed.

SQL> select p.prod_name, s.amount_sold, t.calendar_year
2    from sales s, products p, times t
3    where s.prod_id = p.prod_id
4      and s.time_id = t.time_id
5      and p.prod_id < :pid;

PROD_NAME AMOUNT_SOLD CALENDAR_YEAR
--------- ----------- -------------
...
9 rows selected.

SQL> select p.prod_name, s.amount_sold, t.calendar_year
2    from sales s, products p, times t
3    where s.prod_id = p.prod_id
4      and s.time_id = t.time_id
5      and p.prod_id < :pid;

PROD_NAME AMOUNT_SOLD CALENDAR_YEAR
--------- ----------- -------------
...
9 rows selected.

SQL> alter session set optimizer_capture_sql_plan_baselines = false;

Session altered.



Automatic plan capture will not occur for a statement if a stored outline exists for it and is enabled and the parameter use_stored_outlines is TRUE. In this case, turn on incremental capture of plans into an STS using the function capture_cursor_cache_sqlset() in the DBMS_SQLTUNE package. After you have collected the plans for your workload into the STS, manually create SQL plan baselines using the method described earlier. Then, disable the stored outlines or set use_stored_outlines to FALSE. From now on, SPM will manage your workload and stored outlines will not be used for those statements.

In this article, we have seen how to create SQL plan baselines. In the next, we will describe the SPM aware optimizer and how it uses SQL plan baselines.
06:30 Outerjoins in Oracle (10373 Bytes) » Inside the Oracle Optimizer - Removing the black magic
Since release 6, Oracle has supported a restricted form of left outerjoin, which uses Oracle-specific syntax. In 9i, Oracle introduced support for ANSI SQL 92/99 syntax for inner joins and various types of outerjoin. The Oracle syntax for left outerjoin and that of ANSI SQL 92/99 are not equivalent, as the latter is more expressive.

There appears to be some confusion about equivalence between ANSI outer join and Oracle outer join syntaxes. The following examples explain the equivalences and inequivalences of these two syntaxes.

Oracle-Specific Syntax

Consider query A, which expresses a left outerjoin in the Oracle syntax. Here T1 is the left table whose non-joining rows will be retained and all non-joining rows of T2 will be null appended.


A.
SELECT T1.d, T2.c
FROM T1, T2
WHERE T1.x = T2.x (+);


ANSI Left Outerjoin

In the ANSI outer join syntax, query A can be expressed as query B.


B.
SELECT T1.d, T2.c
FROM T1 LEFT OUTER JOIN T2
ON (T1.x = T2.x);


Equivalence

Consider the following queries. In the Oracle semantics, the presence of (+) on the filter predicate (e.g., T2.y (+) > 5 in query C) indicates that this filter must be applied to the table T2 before the outer join takes place.


C.
SELECT T1.d, T2.c
FROM T1, T2
WHERE T1.x = T2.x (+) and T2.y (+) > 5;


The ANSI left outer join query D is equivalent to C. Applying the filter on the right table in the left outer join is the same as combining the filter with the join condition.


D.
SELECT T1.d, T2.c
FROM T1 LEFT OUTER JOIN T2
ON (T1.x = T2.x and T2.y > 5);


Similarly, the presence of (+) on the filter predicate, T2.y (+) IS NULL, in query E indicates that this filter must be applied to the table T2 before the outer join takes place.


E.
SELECT T1.d, T2.c
FROM T1, T2
WHERE T1.x = T2.x (+) and T2.y (+) IS NULL;


The ANSI left outer join query F is equivalent to E.


F.
SELECT T1.d, T2.c
FROM T1 LEFT OUTER JOIN T2
ON (T1.x = T2.x and T2.y IS NULL);


Consider query G. Oracle will apply the filter, T2.y IS NULL, in query G after the outer join has taken place. G will return only those rows of T2 that failed to join with T1 or those whose T2.y values happen to be null.


G.
SELECT T1.d, T2.c
FROM T1, T2
WHERE T1.x = T2.x (+) and T2.y IS NULL;


The ANSI left outer join query H is equivalent to G, as the WHERE clause in H is applied after the left outer join is performed based on the condition specified in the ON clause.


H.
SELECT T1.d, T2.c
FROM T1 LEFT OUTER JOIN T2
ON (T1.x = T2.x)
WHERE T2.y IS NULL;


Consider query I, where the filter on the left table is applied before or after the outer join takes place.


I.
SELECT T1.d, T2.c
FROM T1, T2
WHERE T1.x = T2.x (+) and T1.Z > 4;


The ANSI left outer join query J is equivalent to query I.


J.
SELECT T1.d, T2.c
FROM T1 LEFT OUTER JOIN T2
ON (T1.x = T2.x)
WHERE T1.Z > 4;


Lateral Views

In Oracle, ANSI left and right outerjoins are internally expressed in terms of left outerjoined lateral views. In many cases, a left outerjoined lateral view can be merged and the ANSI left (or right) outerjoin can be expressed entirely in terms of Oracle native left outerjoin operator. (A lateral view is an inline view that contains correlation referring to other tables that precede it in the FROM clause.)

Consider the ANSI left outer join query K, which is first represented internally as L.


K.
SELECT T1.d, T2.c
FROM T1 LEFT OUTER JOIN T2
ON (T1.x = T2.x and T2.k = 5);

L.
SELECT T1.d, LV.c
FROM T1,
LATERAL (SELECT T2.C
FROM T2
WHERE T1.x = T2.x and T2.k = 5)(+) LV;


The lateral view in query L is merged to yield query M.


M.
SELECT T1.d, T2.c
FROM T1, T2
WHERE T1.x = T2.x (+) and T2.k (+)= 5;


Consider query N, which expresses a left outerjoin in the ANSI join syntax. Currently query N cannot be expressed using the Oracle native left outer join operator.


N.
SELECT T1.m, T2.n
FROM T1 LEFT OUTER JOIN T2
ON (T1.h = 11 and T1.y = T2.y)
WHERE T1.q > 3;


The query N is converted into query O with a left outer-joined lateral view. The lateral view in O cannot be merged, since the filter on the left table specified in the ON clause must be part of the left outerjoin condition.


O.
SELECT T1.m, LV.n
FROM T1,
LATERAL(SELECT T2.n
FROM T2
WHERE T1.h = 11 and T1.y = T2.y)(+) LV
WHERE T1.q > 3;


Consider query P with a disjunction in the outer join condition. Currently N cannot be expressed using the Oracle native left outer join operator.


P.
SELECT T1.A, T2.B
FROM T1 LEFT OUTER JOIN T2
ON (T1.x = T2.x OR T1.Z = T2.Z);


The query P will be converted into Q with a left outer-joined lateral view containing the disjunctive join condition.


Q.
SELECT T1.A, LV.B
FROM T1,
LATERAL (SELECT T2.B
FROM T2
WHERE T1.x = T2.x OR T1.Z = T2.Z) (+) LV;


ANSI Full Outerjoin

Before Oracle 11gR1 all ANSI full outerjoins were converted into a UNION ALL query with two branches, where one branch contained a left outerjoined lateral view and the other branch contained a NOT EXISTS subquery. A native support for hash full outerjoin was introduced in 11gR1 to overcome this problem. When the native full outerjoin, cannot be used, Oracle reverts to the pre-11gR1 strategy.

Consider query R, which specifies an ANSI full outerjoin.


R.
SELECT T1.c, T2.d
FROM T1 FULL OUTER JOIN T2
ON T1.x = T2.y;


Before 11gR1, Oracle would internally convert query R into S.


S.
SELECT T1.c, T2.d
FROM T1, T2
WHERE T1.x = T2.y (+)
UNION ALL
SELECT NULL, T2.d
FROM T2
WHERE NOT EXISTS
(SELECT 1 FROM T1 WHERE T1.x = T2.y);


With the native support of hash full outerjoin, R will be simply expressed as query T, where the view, VFOJ, is considered unmergeable.


T.
SELECT VFOJ.c, VFOJ.d
FROM (SELECT T1.c, T2.d
FROM T1, T2
WHERE T1.x F=F T2.y) VFOJ;


Conversion of Outerjoin into Inner Join

Consider query U. Here the filter predicate on the outer-joined table T2 does not contain the outerjoin operator (+); thus it will be applied after the left outerjoin has taken place. This will result in the elimination of all null appended rows of T2. Hence, Oracle converts the outer join into an inner join.


U.
SELECT T1.d, T2.c
FROM T1, T2
WHERE T1.x = T2.x (+) and T2.y > 5;


The ANSI left outer join query V is equivalent to query U, as the WHERE clause in V is applied after the left outer join is performed based on the condition specified in the ON clause.


V.
SELECT T1.d, T2.c
FROM T1 LEFT OUTER JOIN T2
ON (T1.x = T2.x)
WHERE T2.y > 5;


Oracle converts the queries U and V into query W with an inner join.


W.
SELECT T1.d, T2.c
FROM T1, T2
WHERE T1.x = T2.x and T2.y > 5;



Q&A

Q1: I do not understand the queries N and O. What is the difference between
a filter appearing in the ON Clause or a filter appearing in the WHERE
clause?

A1: Consider two tables T11 and T22.

T11:
A | B
1 | 2
2 | 3
3 | 5

T22:
X | Y
7 | 2
8 | 4
9 | 4

The following ANSI left outer join query N’ involving
the tables T11 and T22 will return three rows, since
the filter, which always fails, is part of the join
condition. Although this join condition, which
comprises both the predicates in the ON clause,
always evaluates to FALSE, all the rows of the left
table T11 are retained in the result.

N’.
SELECT *
FROM T11 LEFT OUTER JOIN T22
ON (T11.A > 9 and T11.B = T22.Y);

A B X Y
------ ---------- ---------- ---------
1 2
2 3
3 5

However, if the filter, T11.A > 9, is moved to the WHERE clause,
the query will return zero rows.


Q2: Is the outer to inner join conversion a new feature?

A2: No. This feature has been avaliable since Release 7.


Q3: Has native full outer join been made available in
versions prior to 11gR1?

A3: Yes. It is available in 10.2.0.3 and 10.2.0.4, but not by
default.
06:30 Oracle keeps closing my TAR because I cannot provide a testcase, can you help? (12361 Bytes) » Inside the Oracle Optimizer - Removing the black magic
The answer to this question is yes, as Oracle Database 11g provides a new diagnostic tool called SQL Test Case Builder. In this article, we explain what SQL Test Case Builder is, and how to use it with examples.

Why SQL Test Case Builder?

For most SQL problems, the single most important factor for a speedy bug resolution is to obtain a reproducible test case. However, this is normally the longest and most painful step for customers. The goal of the SQL Test Case Builder (TCB) is to automatically gather as much information as possible related to a SQL incident (problem) and package it in a way that allows a developer or a support engineer to reproduce the problem on his or her own machine quickly.

At a very high-level, SQL Test Case Builder can be seen as a way to export a SQL. Currently, Oracle export (expdp) takes a schema or a set of tables and exports all the dependents objects. SQL Test Case Builder provides the same service but takes a SQL statement as input.

What's Inside Test Case Builder?

The main input of SQL Test Case Builder is a SQL object. A SQL object is defined as the SQL text plus all the information required to compile it on a particular database instance (this contains the parsing user name, for example).

Logically, a SQL test case appears as a script containing all the necessary commands to recreate the objects, the user, the statistics, and the environment.

Within the Oracle Diagnosability infrastructure, TCB compiles the problem SQL in a special capture mode to obtain the set of objects to export. A test case captures two types of information:

  1. Permanent information

    • SQL text

    • PL/SQL functions, procedures, packages

    • Statistics

    • Bind variables

    • Compilation environment

    • User information (like privileges)

    • SQL profiles, stored outlines, or other SQL Management Objects

    • Meta data on all the objects involved

    • Optimizer statistics

    • The execution plan information

    • The table content (sample or full). This is optional.

  2. Transient information

  3. For most of the SQL test cases, the permanent information above is enough to reproduce a problem. There are however cases where this is not enough and additional information about the context in which this SQL was compiled is required. Therefore, in addition to the permanent information, SQL Test Case Builder captures transient information, e.g. information that is only available as part of the compilation of the SQL statement. This includes dynamic sampling results, cached information, some run time information, like the actual degree of parallelism used, etc.

    As part of creating a SQL test case, the SQL object is reloaded and all the diagnostic information available generated and gathered automatically. This information will be made available to Oracle support and developers.


How do I use the SQL Test Case Builder?

The task of creating a SQL test case can be performed in two ways:
  • From EM (Enterprise Manager), where TCB is invoked on user-demand via IPS (Incident Packaging Service) after a SQL incident occurred. The user can also manually create an incident for a problem query for building test case purpose.

  • From SQLPLUS, where you can directly invoke one of the PL/SQL API functions in the SQL Diagnostic package. We will give examples of using the APIs below.

All the new PL/SQL procedures supporting SQL Test Case Builder are part of a new PL/SQL package called dbms_sqldiag (see dbmsdiag.sql for details). The two main features related to TCB in this package are export and import test cases.

  • Procedure dbms_sqldiag.export_sql_testcase exports a SQL test case for a given SQL statement to a given directory.

  • Procedure dbms_sqldiag.import_sql_testcase imports a test case from a given directory.

To build (or export) a test case, the simplest form would be something like:

     dbms_sqldiag.export_sql_testcase(
directory => 'TCB_DIR_EXP',
sql_text => 'select count(*) from sales',
testcase => tco)

Here directory and sql_text are inputs which specify where the test case will be stored, and the problem query statement, respectively. Testcase specifies the test case metadata as output.

For security reason, the user data are not exported by default. You have the option to set exportData to TRUE to include the data. You can also set samplingPercent if you are exporting with data. To protect users proprietary codes, TCB will not export PL/SQL package body by default.

Once the test case has been built, you can copy all the files under the export directory to your test environment. Note there is a file called xxxxxxxxmain.xml, for example, oratcb1_03C600800001main.xml, which contains the metadata of the test case.

Now importing the test case can be as simple as:


     dbms_sqldiag.import_sql_testcase(
directory => 'TEST_DIR',
filename => 'oratcb1_03C600800001main.xml')

To verify that the test case is successfully rebuilt, you can just issue an explain command for the problem query. However, if you want to actully run the query, then you need to have the data available.

You can refer to dbmsdiag.sql for more information about other options available for these procedures.

Example - We now show the typical steps of using TCB by a sample query with materialized view. In this exmaple, we set the exportData option to TRUE, so we can re-run the same query after the TCB task is completed.

  1. Setup


  2. SQL> connect / as sysdba
    Connected.
    SQL>
    SQL> create or replace directory TCB_DIR_EXP as
    2 '/net/tiger/apps/tcb_exp';
    Directory created.
    SQL>
    SQL> grant dba to apps;
    Grant succeeded.
    SQL>
    SQL> connect apps/apps
    Connected.
    SQL>
    SQL> create materialized view scp_mvu
    2 parallel 2
    3 as
    4 select p.prod_name, c.cust_gender,
    5 max(s.amount_sold) max_amount_sold
    6 from sales s, products p, customers c
    7 where s.prod_id = p.prod_id
    8 and s.cust_id = c.cust_id
    9 group by p.prod_name, c.cust_gender;

    Materialized view created.

    SQL>
    SQL> desc scp_mvu;
    Name Null? Type
    ----------------------------------------- -------- ------------
    PROD_NAME NOT NULL VARCHAR2(50)
    CUST_GENDER CHAR(1)
    MAX_AMOUNT_SOLD NUMBER

    SQL>
    SQL> select * from scp_mvu where max_amount_sold > 7000 order by 3;

    PROD_NAME C MAX_AMOUNT_SOLD
    -------------------------------------------------- - ---------------
    Joseph Sportcoat F 7400.8
    Kenny Cool Leather Skirt M 7708
    Leather Boot-Cut Trousers M 8184

    3 rows selected.

  3. Export as user APPS


  4. SQL> connect apps/apps
    Connected.

    SQL>
    SQL> Rem define the problem SQL statement
    SQL> create or replace package define_vars is
    2 sql_stmt1 varchar2(2000) := q'# select * from scp_mvu
    3 where max_amount_sold > 7000
    4 order by 3
    5 #';
    6 end;
    7 /

    Package created.
    SQL>
    SQL> set serveroutput on
    SQL>
    SQL> declare
    2 tco clob;
    3 begin
    4 -- Export test case
    5 dbms_sqldiag.export_sql_testcase
    6 (
    7 directory => 'TCB_DIR_EXP',
    8 sql_text => define_vars.sql_stmt1,
    9 user_name => 'APPS',
    10 exportData => TRUE,
    11 testcase => tco
    12 );
    13
    14 end;
    15 /

    PL/SQL procedure successfully completed.
    SQL>
    SQL> Rem Drop MV before importing
    SQL> drop materialized view scp_mvu;

    Materialized view dropped.

    At this stage, the export procedure has successfully completed. The next commands prepare a directory for import purpose. The directory could be on a different machine.

    SQL> conn / as sysdba
    Connected.
    SQL> create or replace directory TCB_DIR_IMP
    2 as '/net/lion/test/tcb_imp';
    Directory created.
    SQL>
    SQL> grant dba to test;
    Grant succeeded.

    As the export has finished successfully, you can now transfer all the files under TCB_DIR_EXP to a directory in test environment, for example, TCB_DIR_IMP as created above. Again, look up and make note of the TCB metadata file xxxxxxxxmain.xml, which will be used below.

  5. Import as user TEST


  6. SQL> connect test/test
    Connected.
    SQL>
    SQL> set serveroutput on
    SQL>
    SQL> begin
    2 -- Import test case
    3 dbms_sqldiag.import_sql_testcase
    4 (
    5 directory => 'TCB_DIR_IMP',
    6 filename => 'oratcb3_05e803500001main.xml',
    7 importData => TRUE
    8 );
    9
    10 end;
    11 /

    PL/SQL procedure successfully completed.

  7. Verification. This is to check that now all relevant objects were imported successfully.

SQL> desc scp_mvu;
Name Null? Type
----------------------------------------- -------- ------------
PROD_NAME NOT NULL VARCHAR2(50)
CUST_GENDER CHAR(1)
MAX_AMOUNT_SOLD NUMBER
SQL>
SQL> select * from scp_mvu where max_amount_sold > 7000 order by 3;

PROD_NAME C MAX_AMOUNT_SOLD
-------------------------------------------------- - ---------------
Joseph Sportcoat F 7400.8
Kenny Cool Leather Skirt M 7708
Leather Boot-Cut Trousers M 8184

3 rows selected.

Finally, we also have good news for 10g users: SQL Test Case Builder has been backported to 10.2.0.4!
06:30 Oracle Open World follow up (1065 Bytes) » Inside the Oracle Optimizer - Removing the black magic
We were delighted to see so many people turn up for our Open World session - Inside the 11g Optimizer - so early on Tuesday morning! A lot of people have been asking where they can find more information on the topics covered especially the demos that were shown. You can find similar worked examples for most of the new 11g Optimizer features on the Oracle By Example website. You can also get more information on SQL Plan Management in the following white paper. Our blog entry from December 2007 has more information on Adaptive Cursor Sharing while the January 2008 entry gives more details on the enhancements made to statistics. We hope you enjoyed Oracle Open World as much as we did!
06:30 Oracle Open World 2009 Summary (2403 Bytes) » Inside the Oracle Optimizer - Removing the black magic
We had a great time talking to our users at Open World 2009 both at our Demogrounds booth and at our two sessions. We received a lot of interesting questions during the Optimizer Roundtable discussion, but we did not get to answer all of them due to time constraints. We plan to address the questions we received (both answered and unanswered) in future blog posts... so stay tuned. If you didn't get to attend the discussion, but have a question about the Optimizer, submit it through the email link above.

For those of you who did not get a chance to stop by our Demogrounds booth, here's a recap of the new features that we talked about. Many of the topics have already been covered in earlier blog posts.
These topics are focused on well-known pain points from earlier versions of Oracle. But we also have plenty of new optimizations in Oracle 11gR1 and 11gR2. Stay tuned for details about some of our new optimizations.

06:30 Open World Recap and New White papers (2054 Bytes) » Inside the Oracle Optimizer - Removing the black magic
The Optimizer group has two session and a demo station in the Database campground at this year's Oracle Open World. We will give a technical presentation on What to Expect from the Oracle Optimizer When Upgrading to Oracle Database 11g and host an Oracle Optimizer Roundtable.

The technical session, which is on Tuesday Oct 13 at 2:30 pm, gives step by step instructions on how to use the new 11g features to ensure your upgrade goes smoothly and without any SQL plan regressions. This session is based on our latest white papers, Upgrading from Oracle Database 10g to 11g: What to expect from the Optimizer and SQL Plan Management in Oracle Database 11g.

The roundtable, which is on Thursday Oct. 15th at 10:30 am, will give you a first hand opportunity to pose you burning Optimizer and statistics questions directly to a panel of our leading Optimizer developers. In fact if you plan to attend the roundtable and already know what questions you would like to ask, then please send them to us via email and we will be sure to include them. Other wise, you can hand in your questions at our demo station at any stage during the week, or as you enter the actual session. Just be sure to write your questions in clear block capitals!

We look forward to see you all at Open world.

06:30 Maintaining statistics on large partitioned tables (10962 Bytes) » Inside the Oracle Optimizer - Removing the black magic
We have gotten a lot of questions recently regarding how to gather and maintain optimizer statistics on large partitioned tables. The majority of these questions can be summarized into two topics:

  1. When queries access a single partition with stale or non-existent partition level statistics I get a sub optimal plan due to “Out of Range” values

  2. Global statistics collection is extremely expensive in terms of time and system resources

This article will describe both of these issues and explain how you can address them both in Oracle Database 10gR2 and 11gR1.


Out of Range
Large tables are often decomposed into smaller pieces called partitions in order to improve query performance and ease of data management. The Oracle query optimizer relies on both the statistics of the entire table (global statistics) and the statistics of the individual partitions (partition statistics) to select a good execution plan for a SQL statement. If the query needs to access only a single partition, the optimizer uses only the statistics of the accessed partition. If the query access more than one partition, it uses a combination of global and partition statistics.

“Out of Range” means that the value supplied in a where clause predicate is outside the domain of values represented by the [minimum, maximum] column statistics. The optimizer prorates the selectivity based on the distance between the predicate value and the maximum value (assuming the value is higher than the max), that is, the farther the value is from the maximum value, the lower the selectivity will be. This situation occurs most frequently in tables that are range partitioned by a date column, a new partition is added, and then queried while rows are still being loaded in the new partition. The partition statistics will be stale very quickly due to the continuous trickle feed load even if the statistics get refreshed periodically. The maximum value known to the optimizer is not correct leading to the “Out of Range” condition. The under-estimation of selectivity often leads the query optimizer to pick a sub optimal plan. For example, the query optimizer would pick an index access path while a full scan is a better choice.

The "Out of Range" condition can be prevented by using the new copy table statistics procedure available in Oracle Database10.2.0.4 and 11g. This procedure copies the statistics of the source [sub] partition to the destination [sub] partition. It also copies the statistics of the dependent objects: columns, local (partitioned) indexes etc. It adjusts the minimum and maximum values of the partitioning column as follows; it uses the high bound partitioning value as the maximum value of the first partitioning column (it is possible to have concatenated partition columns) and high bound partitioning value of the previous partition as the minimum value of the first partitioning column for range partitioned table. It can optionally scale some of the other statistics like the number of blocks, number of rows etc. of the destination partition.

Assume we have a table called SALES that is ranged partitioned by quarter on the SALES_DATE column. At the end of every day data is loaded into latest partition. However, statistics are only gathered at the end of every quarter when the partition is fully loaded. Assuming global and partition level statistics (for all fully loaded partitions) are up to date, use the following steps in order to prevent getting a sub-optimal plan due to “out of range”.


  1. Lock the table statistics using LOCK_TABLE_STATS procedure in DBMS_STATS. This is to avoid interference from auto statistics job.


    EXEC DBMS_STATS.LOCK_TABLE_STATS('SH','SALES');


  2. Before beginning the initial load into each new partition (say SALES_Q4_2000) copy the statistics from the previous partition (say SALES_Q3_2000) using COPY_TABLE_STATS. You need to specify FORCE=>TRUE to override the statistics lock.

    EXEC DBMS_STATS.COPY_TABLE_STATS ('SH', 'SALES', 'SALES_Q3_2000', 'SALES_Q4_2000', FORCE=>TRUE);


Expensive global statistics collection

In data warehouse environment it is very common to do a bulk load directly into one or more empty partitions. This will make the partition statistics stale and may also make the global statistics stale. Re-gathering statistics for the effected partitions and for the entire table can be very time consuming. Traditionally, statistics collection is done in a two-pass approach:


  • In the first pass we will scan the table to gather the global statistics

  • In the second pass we will scan the partitions that have been changed to gather their partition level statistics.

The full scan of the table for global statistics collection can be very expensive depending on the size of the table. Note that the scan of the entire table is done even if we change a small subset of partitions.

In Oracle Database 11g, we avoid scanning the whole table when computing global statistics by deriving the global statistics from the partition statistics. Some of the statistics can be derived easily and accurately from partition statistics. For example, number of rows at global level is the sum of number of rows of partitions. Even global histogram can be derived from partition histograms. But the number of distinct values (NDV) of a column cannot be derived from partition level NDVs. So, Oracle maintains another structure called a synopsis for each column at the partition level. A synopsis can be considered as sample of distinct values. The NDV can be accurately derived from synopses. We can also merge multiple synopses into one. The global NDV is derived from the synopsis generated by merging all of the partition level synopses. To summarize


  1. Gather statistics and create synopses for the changed partitions only
  2. Oracle automatically merges partition level synopses into a global synopsis
  3. The global statistics are automatically derived from the partition level statistics and global synopses


Incremental maintenance feature is disabled by default. It can be enabled by changing the INCREMENTAL table preference to true. It can also be enabled for a particular schema or at the database level. If you are interested in more details of the incremental maintenance feature, please refer to the following paper presented in SIGMOD 2008 and to our previous blog entry on new ndv gathering in 11g.


Assume we have table called SALES that is range partitioned by day on the SALES_DATE column. At the end of every day data is loaded into latest partition and partition statistics are gathered. Global statistics are only gathered at the end of every month because gathering them is very time and resource intensive. Use the following steps in order to maintain global statistics after every load.

  1. Turn on incremental feature for the table.


    EXEC DBMS_STATS.SET_TABLE_PREFS('SH','SALES','INCREMENTAL','TRUE');

  2. At the end of every load gather table statistics using GATHER_TABLE_STATS command. You don’t need to specify the partition name. Also, do not specify the granularity parameter. The command will collect statistics for partitions with stale or missing statistics and update the global statistics based on the partition level statistics and synopsis.


    EXEC DBMS_STATS.GATHER_TABLE_STATS('SH','SALES');


Note: that the incremental maintenance feature was introduced in Oracle Database 11g Release 1. However, we also provide a solution in Oracle Database10g Release 2 (10.2.0.4) that simulates the same behavior. The 10g solution is a new value, 'APPROX_GLOBAL AND PARTITION' for the GRANULARITY parameter of the GATHER_TABLE_STATS procedures. It behaves the same as the incremental maintenance feature except that we don’t update the NDV for non-partitioning columns and number of distinct keys of the index at the global level. For partitioned column we update the NDV as the sum of NDV at the partition levels. Also we set the NDV of columns of unique indexes as the number of rows of the table. In general, non-partitioning column NDV at the global level becomes stale less often. It may be possible to collect global statistics less frequently then the default (when table changes 10%) since approx_global option maintains most of the global statistics accurately.

Let's take a look at an example to see how you would effectively use the Oracle Database 10g approach.

After the data load is complete, gather statistics using DBMS_STATS.GATHER_TABLE_STATS for the last partition (say SALES_11FEB2009), specify granularity => 'APPROX_GLOBAL AND PARTITION'. It will collect statistics for the specified partition and derive global statistics from partition statistics (except for NDV as described before).

EXEC DBMS_STATS.GATHER_TABLE_STATS ('SH', 'SALES', 'SALES_11FEB2009', GRANULARITY => 'APPROX_GLOBAL AND PARTITION');

It is necessary to install the one off patch for bug 8719831 if you are using copy_table_stats procedure or APPROX_GLOBAL option in 10.2.0.4 (patch 8877245) or in 11.1.0.7 (patch 8877251).

06:30 Improvement of AUTO sampling statistics gathering feature in Oracle 11g (5970 Bytes) » Inside the Oracle Optimizer - Removing the black magic
Optimizer statistics in Oracle are managed via a pl/sql package, dbms_stats. It provides several pl/sql procedures to gather statistics for a table, schema, or a database. For example, gather_table_statistics is used to gather statistics on a table. This procedure has an estimate_percent parameter, which specifies the sampling percentage of the statistics gathering. The users can specify any number between 0 ~ 100 for this parameter. For example, suppose you have a table BIGT, you can specify a 1% sampling percentage as follows:

exec dbms_stats.gather_table_stats(null, 'BIGT', 
estimate_percent => 1);


It is not always easy for users to pick the appropriate sampling percentage. If the specified sampling percentage is too high, it can take longer to gather statistics. On the contray, if the data is skewed and the specified sampling percentage is too low, the resulting statistics can be inaccurate.

For this reason, Oracle introduced the AUTO value for the estimate_percent parameter. For example, you can gather statistics on BIGT as follows:

exec dbms_stats.gather_table_stats(null, 'BIGT', 
estimate_percent => dbms_stats.auto_sample_size);


The advantage of using AUTO sample size over a fixed number is two-folds. First, when AUTO sample size is specified, the system automatically determines the appropriate sampling percentage. Second, AUTO sample size is more flexible than a fixed sampling percentage. A fixed sampling percentage size that was good at some point in time may not be appropriate after the data distribution in the table has changed. On the other hand when AUTO value is used Oracle will adjust the sample size when the data distribution changes.

When AUTO is used Oracle picks a sample size where the statistics quality is good enough. However, it does not work very well under cases of extreme skew in the data. In Oracle 11g, we improved the behavior when the AUTO value is used. First, AUTO sampling now generates deterministic statistics. Second, and more importantly, AUTO sampling generates statistics that are almost as accurate as 100% sampling but takes much less time than 100% sampling. To illustrate these merits, we compare the performance of using a fixed sampling percentage, AUTO sample size in Oracle 10g and AUTO sample size in Oracle 11g.

We used the standard TPC-D data generator to generate a Lineitem table. The Lineitem table is about 230G and contains 1.8 million rows with 16 columns. The schema of the lineitem table is as follows:

column namecolumn type
l_shipdatedate
l_orderkeynumber
l_discountnumber
l_extendedpricenumber
l_suppkeynumber
l_quantitynumber
l_returnflagvarchar2
l_partkeynumber
l_linestatusvarchar2
l_taxnumber
l_commitdatedate
l_receiptdatedate
l_shipmodevarchar2
l_linenumbernumber
l_shipinstructvarchar2
l_commentvarchar2

Table 1 gives the elapsed time of gathering statistics on the Lineitem table by different sampling percentages.

Sampling PercentageElapsed Time (sec)
1% sampling 797
100% sampling (Compute)18772
Auto sampling in Oracle 10g 2935
Auto sampling in Oracle 11g 1908


Table 1: Statistics gathering time on 230G TPC-D Lineitem Table Using Different Estimate Percentages

We also compare the quality of the statistics gathered using different estimate percentages. Among all the statistics of a column, number of distinct values (NDV) is the one whose accuracy used to be an issue. We define the accuracy rate of NDV of a column as follows:


accuracy rate = 1 - (|estimated NDV - actual NDV|)/actual NDV.


The accuracy rate ranges from 0% to 100%. The higher the accuracy rate is, the more accurate the gathered statistics are. Since 100% sampling always lead to an accuracy rate of 100%, we do not report it. We focus on the columns which has at least one statistics accuracy rate below 99.9% when using different estimate percentages. Table 2 illustrates the accurate rates of the columns.


Column NameActual NDV Auto Sampling in Oracle 11g 1% Sampling
orderkey 450,000,000 98.0% 50%
comment 181,122,127 98.60% 4.60%
partkey 60,000,000 99.20% 98.20%
suppkey 3,000,000 99.60% 99.90%
extendedprice 3,791,320 99.60% 94.30%


Table 2: Accuracy Rate of Gathering NDV LineItem Using Different Estimate Percentages

In short, the elapsed time of ''AUTO sampling in Oracle 11g'' is 10 times faster than 100% sampling but yields the statistics of similar quality (accuracy rate close to 100%).


06:30 Explain adaptive cursor sharing behavior with cursor_sharing = similar and force. (280 Bytes) » Inside the Oracle Optimizer - Removing the black magic
06:30 Displaying and reading the execution plans for a SQL statement (16944 Bytes) » Inside the Oracle Optimizer - Removing the black magic
Generating and displaying the execution plan of a SQL statement is a common task for most DBAs, SQL developers, and preformance experts as it provides them information on the performance characteristics of a SQL statement. An execution plan shows the detailed steps necessary to execute a SQL statement. These steps are expressed as a set of database operators that consumes and produces rows. The order of the operators and their implentation is decided by the query optimizer using a combination of query transformations and physical optimization techniques.

While the display is commonly shown in a tabular format, the plan is in fact tree-shaped. For example, consider the following query based on the SH schema (Sales History):



select prod_category, avg(amount_sold)
from sales s, products p
where p.prod_id = s.prod_id
group by prod_category;


The tabular representation of this query's plan is:



------------------------------------------
Id Operation Name
------------------------------------------
0 SELECT STATEMENT
1 HASH GROUP BY
2 HASH JOIN
3 TABLE ACCESS FULL PRODUCTS
4 PARTITION RANGE ALL
5 TABLE ACCESS FULL SALES
------------------------------------------

While the tree-shaped representation of the plan is:

GROUP BY
|
JOIN
_____|_______
| |
ACCESS ACCESS
(PRODUCTS) (SALES)




When you read a plan tree you should start from the bottom up. In the above example begin by looking at the access operators (or the leaves of the tree). In this case the access operators are implemented using full table scans. The rows produced by these tables scans will be consumed by the join operator. Here the join operator is a hash-join (other alternatives include nested-loop or sort-merge join). Finally the group-by operator implemented here using hash (alternative would be sort) consumes rows produced by the join-opertor.

The execution plan generated for a SQL statement is just one of the many alternative execution plans considered by the query optimizer. The query optimizer selects the execution plan with the lowest cost. Cost is a proxy for performance, the lower is the cost the better is the performance. The cost model used by the query optimizer accounts for the IO, CPU, and network usage in the query.

There are two different methods you can use to look at the execution plan of a SQL statement:


  1. EXPLAIN PLAN command - This displays an execution plan for a SQL statement without actually executing the statement.

  2. V$SQL_PLAN - A dictionary view introduced in Oracle 9i that shows the execution plan for a SQL statement that has been compiled into a cursor in the cursor cache.


Under certain conditions the plan shown when using EXPLAIN PLAN can be different from the plan shown using V$SQL_PLAN. For example, when the SQL statement contains bind variables the plan shown from using EXPLAIN PLAN ignores the bind variable values while the plan shown in V$SQL_PLAN takes the bind variable values into account in the plan generation process.

Displaying an execution plan has been made easier after the introduction of the dbms_xplan package in Oracle 9i and by the enhancements made to it in subsequent releases. This packages provides several PL/SQL procedures to display the plan from different sources:


  1. EXPLAIN PLAN command

  2. V$SQL_PLAN

  3. Automatic Workload Repository (AWR)

  4. SQL Tuning Set (STS)

  5. SQL Plan Baseline (SPM)


The following examples illustrate how to generate and display an execution plan for our original SQL statement using the different functions provided in the dbms_xplan package.

Example 1 Uses the EXPLAIN PLAN command and the dbms_xplan.display function.


SQL> EXPLAIN PLAN FOR
2 select prod_category, avg(amount_sold)
3 from sales s, products p
4 where p.prod_id = s.prod_id
5 group by prod_category;

Explained.



SQL> select plan_table_output
2 from table(dbms_xplan.display('plan_table',null,'basic'));

------------------------------------------
Id Operation Name
------------------------------------------
0 SELECT STATEMENT
1 HASH GROUP BY
2 HASH JOIN
3 TABLE ACCESS FULL PRODUCTS
4 PARTITION RANGE ALL
5 TABLE ACCESS FULL SALES
------------------------------------------


The arguments are for dbms_xplan.display are:


  • plan table name (default 'PLAN_TABLE'),

  • statement_id (default null),

  • format (default 'TYPICAL')


More details can be found in $ORACLE_HOME/rdbms/admin/dbmsxpln.sql.

Example 2 Generating and displaying the execution plan for the last SQL statement executed in a session:



SQL> select prod_category, avg(amount_sold)
2 from sales s, products p
3 where p.prod_id = s.prod_id
4 group by prod_category;

no rows selected


SQL> select plan_table_output
2 from table(dbms_xplan.display_cursor(null,null,'basic'));

------------------------------------------
Id Operation Name
------------------------------------------
0 SELECT STATEMENT
1 HASH GROUP BY
2 HASH JOIN
3 TABLE ACCESS FULL PRODUCTS
4 PARTITION RANGE ALL
5 TABLE ACCESS FULL SALES
------------------------------------------


The arguments used by dbms_xplay.dispay_cursor are:


  • SQL ID (default null, null means the last SQL statement executed in this session),

  • child number (default 0),

  • format (default 'TYPICAL')


The details are in $ORACLE_HOME/rdbms/admin/dbmsxpln.sql.


Example 3 Displaying the execution plan for any other statement requires the SQL ID to be provided, either directly or indirectly:

  1. Directly:

    SQL> select plan_table_output from
    2 table(dbms_xplan.display_cursor('fnrtqw9c233tt',null,'basic'));


  2. Indirectly:

    SQL> select plan_table_output
    2 from v$sql s,
    3 table(dbms_xplan.display_cursor(s.sql_id,
    4 s.child_number, 'basic')) t
    5 where s.sql_text like 'select PROD_CATEGORY%';


Example 4 - Displaying an execution plan corresponding to a SQL Plan Baseline. SQL Plan Baselines have been introduced in Oracle 11g to support the SQL Plan Management feature (SPM). In order to illustrate such a case we need to create a SQL Plan Baseline first.


SQL> alter session set optimizer_capture_sql_plan_baselines=true;

Session altered.

SQL> select prod_category, avg(amount_sold)
2 from sales s, products p
3 where p.prod_id = s.prod_id
4 group by prod_category;

no rows selected

If the above statement has been executed more than once, a SQL Plan Baseline will be created for it and you can verified this using the follows query:


SQL> select SQL_HANDLE, PLAN_NAME, ACCEPTED
2 from dba_sql_plan_baselines
3 where sql_text like 'select prod_category%';

SQL_HANDLE PLAN_NAME ACC
------------------------------ ------------------------------ ---
SYS_SQL_1899bb9331ed7772 SYS_SQL_PLAN_31ed7772f2c7a4c2 YES


The execution plan for the SQL Plan Baseline created above can be displayed either directly or indirectly:

  1. Directly
    select t.* from
    table(dbms_xplan.display_sql_plan_baseline('SYS_SQL_1899bb9331ed7772',
    format => 'basic')) t


  2. Indirectly
    select t.*
    from (select distinct sql_handle
    from dba_sql_plan_baselines
    where sql_text like 'select prod_category%') pb,
    table(dbms_xplan.display_sql_plan_baseline(pb.sql_handle,
    null,'basic')) t;



The output of either of these two statements is:



----------------------------------------------------------------------------
SQL handle: SYS_SQL_1899bb9331ed7772
SQL text: select prod_category, avg(amount_sold) from sales s, products p
where p.prod_id = s.prod_id group by prod_category
----------------------------------------------------------------------------

----------------------------------------------------------------------------
Plan name: SYS_SQL_PLAN_31ed7772f2c7a4c2
Enabled: YES Fixed: NO Accepted: YES Origin: AUTO-CAPTURE
----------------------------------------------------------------------------

Plan hash value: 4073170114

---------------------------------------------------------
Id Operation Name
---------------------------------------------------------
0 SELECT STATEMENT
1 HASH GROUP BY
2 HASH JOIN
3 VIEW index$_join$_002
4 HASH JOIN
5 INDEX FAST FULL SCAN PRODUCTS_PK
6 INDEX FAST FULL SCAN PRODUCTS_PROD_CAT_IX
7 PARTITION RANGE ALL
8 TABLE ACCESS FULL SALES
---------------------------------------------------------


Formatting


The format argument is highly customizable and allows you to see as little (high-level) or as much (low-level) details as you need / want in the plan output. The high-level options are:

  1. Basic
    The plan includes the operation, options, and the object name (table, index, MV, etc)
  2. Typical
    It includes the information shown in BASIC plus additional optimizer-related internal information such as cost, size, cardinality, etc. These information are shown for every operation in the plan and represents what the optimizer thinks is the operation cost, the number of rows produced, etc. It also shows the predicates evaluation by the operation. There are two types of predicates: ACCESS and FILTER. The ACCESS predicates for an index are used to fetch the relevant blocks because they apply to the search columns. The FILTER predicates are evaluated after the blocks have been fetched.
  3. All
    It includes the information shown in TYPICAL plus the lists of expressions (columns) produced by every operation, the hint alias and query block names where the operation belongs. The last two pieces of information can be used as arguments to add hints to the statement.
The low-level options allow the inclusion or exclusion of find details, such as predicates and cost.
For example,


select plan_table_output
from table(dbms_xplan.display('plan_table',null,'basic +predicate +cost'));

-------------------------------------------------------
Id Operation Name Cost (%CPU)
-------------------------------------------------------
0 SELECT STATEMENT 17 (18)
1 HASH GROUP BY 17 (18)
* 2 HASH JOIN 15 (7)
3 TABLE ACCESS FULL PRODUCTS 9 (0)
4 PARTITION RANGE ALL 5 (0)
5 TABLE ACCESS FULL SALES 5 (0)
-------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("P"."PROD_ID"="S"."PROD_ID")



select plan_table_output from
table(dbms_xplan.display('plan_table',null,'typical -cost -bytes'));

----------------------------------------------------------------------------
Id Operation Name Rows Time Pstart Pstop
----------------------------------------------------------------------------
0 SELECT STATEMENT 4 00:00:01
1 HASH GROUP BY 4 00:00:01
* 2 HASH JOIN 960 00:00:01
3 TABLE ACCESS FULL PRODUCTS 766 00:00:01
4 PARTITION RANGE ALL 960 00:00:01 1 16
5 TABLE ACCESS FULL SALES 960 00:00:01 1 16
----------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("P"."PROD_ID"="S"."PROD_ID")

Note Section


In addition to the plan, the package displays notes in the NOTE section, such as that dynamic sampling was used during query optimization or that star transformation was applied to the query.
For example, if the table SALES did not have statistics then the optimizer will use dynamic sampling and the plan display will report it as follows (see '+note' detail in the query):


select plan_table_output
from table(dbms_xplan.display('plan_table',null,'basic +note'));

------------------------------------------
Id Operation Name
------------------------------------------
0 SELECT STATEMENT
1 HASH GROUP BY
2 HASH JOIN
3 TABLE ACCESS FULL PRODUCTS
4 PARTITION RANGE ALL
5 TABLE ACCESS FULL SALES
------------------------------------------

Note
-----
- dynamic sampling used for this statement


Bind peeking



The query optimizer takes into account the values of bind variable values when generation an execution plan. It does what is generally called bind peeking. See the first post in this blog about the concept of bind peeking and its impact on the plans and the performance of SQL statements.
As stated earlier the plan shown in V$SQL_PLAN takes into account the values of bind variables while the one shown from using EXPLAIN PLAN does not. Starting with 10gR2, the dbms_xplan package allows the display of the bind variable values used to generate a particular cursor/plan. This is done by adding '+peeked_binds' to the format argument when using display_cursor().
This is illustrated with the following example:



variable pcat varchar2(50)
exec :pcat := 'Women'

select PROD_CATEGORY, avg(amount_sold)
from sales s, products p
where p.PROD_ID = s.PROD_ID
and prod_category != :pcat
group by PROD_CATEGORY;

select plan_table_output
from table(dbms_xplan.display_cursor(null,null,'basic +PEEKED_BINDS'));

------------------------------------------
Id Operation Name
------------------------------------------
0 SELECT STATEMENT
1 HASH GROUP BY
2 HASH JOIN
3 TABLE ACCESS FULL PRODUCTS
4 PARTITION RANGE ALL
5 TABLE ACCESS FULL SALES
------------------------------------------

Peeked Binds (identified by position):
--------------------------------------

1 - :PCAT (VARCHAR2(30), CSID=2): 'Women'
06:30 11gr2 IGNORE_ROW_ON_DUPKEY_INDEX hint bug (2474 Bytes) » Brotherxiao's Home

детско обзавеждане

Oracle 11gR2 New feature中引入了IGNORE_ROW_ON_DUPKEY_INDEX hint以解决insert …select 碰到的唯一性冲突问题,测试了一把,没有成功。看来新功能还有带检验。
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning and Real Application Testing options
SQL> show user
USER is "SYS"
SQL> create table t1 as select * from dba_objects;
Table created.
SQL> select count(*) from t1;
COUNT(*)
----------
65690
SQL> insert into t1 select * from t1 where rownum<10000;
9999 rows created.
SQL> commit;
Commit complete.
SQL> select count(*) from t1;
COUNT(*)
----------
75689
SQL> create table t2 as select * from t1 where 1=0;
Table created.
SQL> alter table t2 add primary key(object_id);
Table altered.
SQL> insert into t2 select * from t1;
insert into t2 select * from t1
*
ERROR at line 1:
ORA-00001: unique constraint (SYS.SYS_C007007) violated
SQL> insert /*+ IGNORE_ROW_ON_DUPKEY_INDEX */ into t2 select * from t1;
insert /*+ IGNORE_ROW_ON_DUPKEY_INDEX */ into t2 select * from t1
*
ERROR at line 1:
ORA-00001: unique constraint (SYS.SYS_C007007) violated
SQL> insert /*+ IGNORE_ROW_ON_DUPKEY_INDEX(t2,SYS_C007007) */ into t2
select * from t1;
insert /*+ IGNORE_ROW_ON_DUPKEY_INDEX(t2,SYS_C007007) */ into t2 select * from t1
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [qerltcInsertSelectRop_bad_state],
[], [], [], [], [], [], [], [], [], [], []
SQL> insert into t2 select * from dba_objects;
65692 rows created.
SQL> commit;
Commit complete.
SQL> insert /*+ IGNORE_ROW_ON_DUPKEY_INDEX(t2,SYS_C007007) */ into t2
select * from t1;
insert /*+ IGNORE_ROW_ON_DUPKEY_INDEX(t2,SYS_C007007) */ into t2
select * from t1
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [qerltcInsertSelectRop_bad_state],

[], [], [], [], [], [], [], [], [], [], []
06:30 比 F4 还红的四人组合 (987 Bytes) » Hey!! Sky!

四大天王

       没错,比 F4 还红,不知道 F4 在 40 多岁时会不会像他们这样红。在优酷上看到的这段四人同台的视频,不知道具体是哪一年的,不过从演唱的歌曲和郭富城的头型猜测,应该不会晚于 1994 年看看他们当年的表演吧,哈哈。

 

 

 

 

 华仔的帅气,学友的歌声,郭富城的舞技,黎明的……黎明的……黎明的身高!完美的搭配:)

06:30 根据指定栏位连接两个文件——join 命令 (5660 Bytes) » Hey!! Sky!

       在 linux/unix 中可以使用 join 命令来连接两个文件。它会根据指定栏位,找到两个文件中指定栏位内容相同的行,将他们合并,并根据要求的格式输出内容。该命令对于比较两个文件的内容很有帮助。

语法
join [ -i ][ -a filenumber | -v filenumber ]  [  -1 fieldnumber  ]
[  -2 fieldnumber ]  [ -o list ]  [ -e string ]  [ -t char ]
     file1  file2

主要参数

-i 或 --igore-case   比较栏位内容时,忽略大小写的差异。
-a <1或2>   除了显示原来的输出内容之外,还显示指令文件中没有相同栏位的行。
-v <1或2>   跟-a相同,但是只显示文件中没有相同栏位的行。
-1/-j1 <栏位>   连接[文件1]指定的栏位。栏位从 1 开始,默认为1。
-2/-j2 <栏位>   连接[文件2]指定的栏位。栏位从 1 开始,默认为1。
-j <栏位> 相当于 -1 <栏位> -2 <栏位>
-e <字符串>   若[文件1]与[文件2]中找不到指定的栏位,则在输出中填入选项中的字符串。
-o <格式>   按照指定的格式来显示结果。
-t <字符>   使用栏位的分隔字符。
 --help   显示帮助。
--version   显示版本信息。

实例

oracle DBALNP01@lonespcmp1 > cat timestamp_list_LNWASP1_pre.out.es
Feb 22 14:53 /data/oracle/LNWASP1/data1/LNWASP1_QUEST_SMALL1_DATA_01.dbf
Feb 22 14:53 /data/oracle/LNWASP1/data1/LNWASP1_audits01.dbf
Feb 22 14:54 /data/oracle/LNWASP1/data1/LNWASP1_data01_01.dbf
Feb 22 14:54 /data/oracle/LNWASP1/data1/LNWASP1_index01_01.dbf
Feb 22 14:53 /data/oracle/LNWASP1/data1/LNWASP1_tools_01.dbf
Feb 22 14:53 /data/oracle/LNWASP1/data1/LNWASP1_users_01.dbf
Feb 22 14:53 /data/oracle/LNWASP1/index1/LNWASP1_sysaux02.dbf
Feb 22 14:53 /data/oracle/LNWASP1/index1/LNWASP1_sysaux03.dbf
Feb 22 14:54 /data/oracle/LNWASP1/redo1/LNWASP1_ctl_01.dbf
Feb 22 14:54 /data/oracle/LNWASP1/redo2/LNWASP1_ctl_02.dbf
Feb 22 14:54 /data/oracle/LNWASP1/redo3/LNWASP1_ctl_03.dbf
Feb 22 14:54 /data/oracle/LNWASP1/rollback/LNWASP1_undotbs01.dbf
Feb 22 14:53 /data/oracle/LNWASP1/system/LNWASP1_sysaux01.dbf
Feb 22 14:53 /data/oracle/LNWASP1/system/LNWASP1_system01.dbf
Feb 22 05:06 /data/oracle/LNWASP1/temp/LNWASP1_temp01.dbf
(/apps/oracle/scripts/ADHOC/DOBCPFILETSCHECK/tmp)
oracle DBALNP01@lonespcmp1 > cat timestamp_list_LNWASP1_post.out.es
Feb 22 14:57 /data/oracle/LNWASP1/data1/LNWASP1_QUEST_SMALL1_DATA_01.dbf
Feb 22 14:57 /data/oracle/LNWASP1/data1/LNWASP1_audits01.dbf
Feb 22 14:58 /data/oracle/LNWASP1/data1/LNWASP1_data01_01.dbf
Feb 22 14:58 /data/oracle/LNWASP1/data1/LNWASP1_index01_01.dbf
Feb 22 14:57 /data/oracle/LNWASP1/data1/LNWASP1_tools_01.dbf
Feb 22 14:57 /data/oracle/LNWASP1/data1/LNWASP1_users_01.dbf
Feb 22 14:57 /data/oracle/LNWASP1/index1/LNWASP1_sysaux02.dbf
Feb 22 14:57 /data/oracle/LNWASP1/index1/LNWASP1_sysaux03.dbf
Feb 22 14:58 /data/oracle/LNWASP1/redo1/LNWASP1_ctl_01.dbf
Feb 22 14:58 /data/oracle/LNWASP1/redo2/LNWASP1_ctl_02.dbf
Feb 22 14:58 /data/oracle/LNWASP1/redo3/LNWASP1_ctl_03.dbf
Feb 22 14:58 /data/oracle/LNWASP1/rollback/LNWASP1_undotbs01.dbf
Feb 22 14:57 /data/oracle/LNWASP1/system/LNWASP1_sysaux01.dbf
Feb 22 14:57 /data/oracle/LNWASP1/system/LNWASP1_system01.dbf
Feb 22 05:06 /data/oracle/LNWASP1/temp/LNWASP1_temp01.dbf
(/apps/oracle/scripts/ADHOC/DOBCPFILETSCHECK/tmp)
oracle DBALNP01@lonespcmp1 > join -1 4 -2 4 -o 1.1 1.2 1.3 2.1 2.2 2.3 2.4 timestamp_list_LNWASP1_pre.out.es timestamp_list_LNWASP1_post.out.es
Feb 22 14:53 Feb 22 14:57 /data/oracle/LNWASP1/data1/LNWASP1_QUEST_SMALL1_DATA_01.dbf
Feb 22 14:53 Feb 22 14:57 /data/oracle/LNWASP1/data1/LNWASP1_audits01.dbf
Feb 22 14:54 Feb 22 14:58 /data/oracle/LNWASP1/data1/LNWASP1_data01_01.dbf
Feb 22 14:54 Feb 22 14:58 /data/oracle/LNWASP1/data1/LNWASP1_index01_01.dbf
Feb 22 14:53 Feb 22 14:57 /data/oracle/LNWASP1/data1/LNWASP1_tools_01.dbf
Feb 22 14:53 Feb 22 14:57 /data/oracle/LNWASP1/data1/LNWASP1_users_01.dbf
Feb 22 14:53 Feb 22 14:57 /data/oracle/LNWASP1/index1/LNWASP1_sysaux02.dbf
Feb 22 14:53 Feb 22 14:57 /data/oracle/LNWASP1/index1/LNWASP1_sysaux03.dbf
Feb 22 14:54 Feb 22 14:58 /data/oracle/LNWASP1/redo1/LNWASP1_ctl_01.dbf
Feb 22 14:54 Feb 22 14:58 /data/oracle/LNWASP1/redo2/LNWASP1_ctl_02.dbf
Feb 22 14:54 Feb 22 14:58 /data/oracle/LNWASP1/redo3/LNWASP1_ctl_03.dbf
Feb 22 14:54 Feb 22 14:58 /data/oracle/LNWASP1/rollback/LNWASP1_undotbs01.dbf
Feb 22 14:53 Feb 22 14:57 /data/oracle/LNWASP1/system/LNWASP1_sysaux01.dbf
Feb 22 14:53 Feb 22 14:57 /data/oracle/LNWASP1/system/LNWASP1_system01.dbf
Feb 22 05:06 Feb 22 05:06 /data/oracle/LNWASP1/temp/LNWASP1_temp01.dbf

参考

http://www.linuxmanpages.com/man1/join.1.php
http://www.allwiki.com/index.php?title=Linux%E5%91%BD%E4%BB%A4:Join&variant=zh-cn

06:30 在 vi 中替换字符串 (1060 Bytes) » Hey!! Sky!

       vi/vim 中可以使用 :s 命令来替换字符串。以前只会使用一种格式来全文替换,今天发现该命令有很多种写法(vi 真是强大啊,还有很多需要学习),记录几种在此,方便以后查询。

:s/vivian/sky/ 替换当前行第一个 vivian 为 sky
:s/vivian/sky/g 替换当前行所有 vivian 为 sky

:n,$s/vivian/sky/ 替换第 n 行开始到最后一行中每一行的第一个 vivian 为 sky
:n,$s/vivian/sky/g 替换第 n 行开始到最后一行中每一行所有 vivian 为 sky
n 为数字,若 n 为 .,表示从当前行开始到最后一行

:%s/vivian/sky/(等同于 :g/vivian/s//sky/) 替换每一行的第一个 vivian 为 sky
:%s/vivian/sky/g(等同于 :g/vivian/s//sky/g) 替换每一行中所有 vivian 为 sky

可以使用 # 作为分隔符,此时中间出现的 / 不会作为分隔符
:s#vivian/#sky/# 替换当前行第一个 vivian/ 为 sky/
06:30 口吃、口语、口音 (2835 Bytes) » Hey!! Sky!

       “Hi, Espears DBA, Sky speaking!” 这是进公司将近三个月说得最多的一句英语,每次接 hotline 都要先自报家门。不过这看起来简单的 5 个单词,连起来之后总觉得像绕口令,老是说不清楚,每次说完了之后连自己都想笑,还好老外都比较有礼貌,到现在为止还没有被嘲笑过。感觉越急越口吃,现在干脆一个一个单词慢慢说,说清楚了才是最重要的,哈哈。

       以前常在网上看到一些老外生活中常用口语,短小精干,却很有用。但这些流传的所谓流行口语又有多少人去验证过呢?也不知道是啥年代总结的,现在还流不流行?很少看原版的英语连续剧,太懒了,也没机会从中学到一些。不过进公司几个月来发现两个词英国人(只是英国也许也包括一些英联邦国家,比如澳大利亚,老美用不用就不知道了)用得很多,一个是 cheers,另一个是 cool。Cheers 这个词,中国人都很熟悉,不过也经常误会,cheers 是举杯敬酒之意,但并不一定要干杯,也许碰杯更合适一些,而 bottom up 和中文的干杯比较接近。不过,英国人在平时经常说 cheers,不要以为他们都好酒,其实在英国英语中 cheers 有感谢或者再见的意思,我之前百思不得其解,怎么每次电话会议结束时,那几个老外都要说 cheers,后来看了 Todd 关于 Cheers 的解释总算明白了,哈哈,还好没丢人。至于另一个词 cool 中国人就更加熟悉了,不就是“酷”么,这个词都快被中国人用烂了,我之前以为老外并不会很频繁的用这个词,没想到用的频率不必中国人少,不过意思还是有些差别。当然这里说的肯定不是 cool 的本意,中国人经常用 cool 来形容人外表英俊、帅气,但却有点冷,表情不多,不容易接近,老外好像不用来形容人(不确定有没有,到现在还没听到过),一般都是形容事情进展得顺利,也可以翻译成叹词,太好了、太棒了之类的,反正对于好的事情,说句 cool 肯定没错。

       口音是最让人受不了的,我们的 dba manager 是 Newcastle 的,口音重得要命,我本来听力就烂,再加上一口 Newcastle 英语,要听懂真是难上加难。其中最大的特色就是把 ^ 发成 u,比如 but [b^t] 发成 [but],把  run [r^n] 发成 [run],真是闻所未闻,要是没有人告诉我一下恐怕一辈子都听不懂,哈哈。

06:30 又过生日 (839 Bytes) » Hey!! Sky!

       刚刚没头没脑得忙了一阵,抽空上来转转。2007 年的生日就快过去,没有任何节目,和平常没有任何分别,可能唯一的不同就是今天第一次上通宵 night shift。说来也奇怪,开始还有点小兴奋,很早就赶到公司来了,可能第一次上比较新鲜吧,不过熬到现在的确有点累了。今天一大早就起来开始忙活了,下午想补个觉,却一直无法睡着,处于假寐状态。不过现在倒也不觉得困,只是头有点晕。现在的身体状态真的越来越差了,老是觉得疲惫,真的老了,这不又不知不觉地老了一岁。刚看到新闻,侯耀文同志就这样不知不觉地去了,不知道俺啥时候也就这样,平平淡淡地去了……

06:30 亲娘的,水怎么还没开? (252 Bytes) » Hey!! Sky!
Thinking
06:30 exp query 参数的格式 (5761 Bytes) » Hey!! Sky!

昨天需要按照条件导出一个表,再导入另一个数据库,很自然想到了 exp 的 query 参数,本来以为可以很顺利完成,结果 query 参数的格式怎么也写不对,因为时间紧迫,最后通过 dblink 直接插入了另一个数据库。今天回过头来再研究了一下,发现其实也不复杂,按照文档还是能很好的解决的。

希望导出测试表中日期为 2007-09-22 的一条数据:

SQL> select * from skytest;

       SID DT
---------- -----------------
         1 20070913 00:00:00
         2 20070914 00:00:00
         3 20070915 00:00:00
         4 20070916 00:00:00
         5 20070917 00:00:00
         6 20070918 00:00:00
         7 20070919 00:00:00
         8 20070920 00:00:00
         9 20070921 00:00:00
        10 20070922 00:00:00
        12 20070924 00:00:00

11 rows selected.

凭印象按照以下格式运行:

exp / tables=skytest file=test.dmp query=\"where dt=to_date(\'2007-09-22\',\'yyyy-mm-dd\')\"
结果报错:
ksh: syntax error: `(' unexpected

其实文档中讲得很清楚:

exp scott/tiger TABLES=emp QUERY=\"WHERE job=\'SALESMAN\' and sal \<1600\"
Note:
Because the value of the QUERY parameter contains blanks, most operating systems require that the entire strings WHERE job=\'SALESMAN\' and sal\<1600 be placed in double quotation marks or marked as a literal by some method. Operating system reserved characters also need to be preceded by an escape character. See your Oracle operating system-specific documentation for information about special and reserved characters on your system.

所有操作系统保留字符都要使用转义符号,看来 ( ) 也需要转义。

正确的写法:

oracle DBALNP01 > exp / tables=skytest file=test.dmp query=\"where dt=to_date\(\'2007-09-22\',\'yyyy-mm-dd\'\)\"

Export: Release 8.1.7.4.0 - Production on Wed Sep 12 04:30:45 2007

(c) Copyright 2000 Oracle Corporation.  All rights reserved.


Connected to: Oracle8i Enterprise Edition Release 8.1.7.4.0 - Production
With the Partitioning option
JServer Release 8.1.7.4.0 - Production
Export done in US7ASCII character set and US7ASCII NCHAR character set

About to export specified tables via Conventional Path ...
. . exporting table                        SKYTEST          1 rows exported
Export terminated successfully without warnings.

简洁的写法:
如果环境变量设置合适,就不需要使用 to_date 函数了

oracle DBALNP01 > exp / tables=skytest file=test.dmp query=\"where dt=\'20070922\'\"

Export: Release 8.1.7.4.0 - Production on Wed Sep 12 04:25:56 2007

(c) Copyright 2000 Oracle Corporation.  All rights reserved.


Connected to: Oracle8i Enterprise Edition Release 8.1.7.4.0 - Production
With the Partitioning option
JServer Release 8.1.7.4.0 - Production
Export done in US7ASCII character set and US7ASCII NCHAR character set

About to export specified tables via Conventional Path ...
. . exporting table                        SKYTEST          1 rows exported
Export terminated successfully without warnings.

任何 os 平台都适用的方法(推荐):
以上两种方法并不一定适合其他操作系统,不过使用 parfile 就不用担心这些格式问题

oracle DBALNP01 > cat > test.par
tables=skytest
file=test.dmp
query="where dt=to_date('2007-09-22','yyyy-mm-dd')"

oracle DBALNP01 > exp / parfile=test.par

Export: Release 8.1.7.4.0 - Production on Wed Sep 12 04:22:27 2007

(c) Copyright 2000 Oracle Corporation.  All rights reserved.


Connected to: Oracle8i Enterprise Edition Release 8.1.7.4.0 - Production
With the Partitioning option
JServer Release 8.1.7.4.0 - Production
Export done in US7ASCII character set and US7ASCII NCHAR character set

About to export specified tables via Conventional Path ...
. . exporting table                        SKYTEST          1 rows exported
Export terminated successfully without warnings.

不需要任何转义符既简洁又可以在多种操作系统平台上通用,推荐使用这种方式。

参考:http://www.itpub.net/683793.html

06:30 Where Amazing Happens (2151 Bytes) » Hey!! Sky!
      终于,火箭的连胜在今天被全联盟排名第一的凯尔特人终结,而且是大比分败下阵来。但这不影响球迷对火箭的喜爱,是时候该歇一歇为后面的常规赛甚至是季后赛做更好的准备了。

      22 连胜,你们已经创造了奇迹,也许还会再继续创造奇迹。 Where Amazing Happens ...

关于 Where Amazing Happens

      本赛季宣传片的配乐是一段淡雅疏朗的钢琴曲,令人倍感清新。这段曲子出自卡利·卡曼多(Carly Comando)之手,她是一位来自纽约布鲁克林的24岁女孩。实际上,这段钢琴曲并不是特意为NBA创造,而是一段网络视频的配乐。

      该视频名为《每天》(Everyday),由一个名叫诺阿·卡利纳(Noah Kalina)的小伙子完成。他连续六年每天拍一张照片,然后把所有照片按照幻灯片的形式,浓缩在5分钟的视频内快速播放。这段视频在网络上名声大振,卡曼多主动为其配乐。



      这位仁兄还在继续着他每天拍一张照片的 project,可以去看看他这个 project 的主页

      另外,关于 Where Amazing Happens 的视频网上还有很多,不管是官方的还是网友自己制作的,有些挺搞笑的,可以去 youku 上看看
06:30 UNIX 的时间和时区 (3099 Bytes) » Hey!! Sky!

时间
       所有 UNIX 系统都采用格林威治时间从1970-01-01午夜后开始计算的微秒数来保存时间,这一日期通常称为 epoch,正值表示为 1970 以后,负值则表示 1970 年以前。
       时间刷新频率由 /usr/include/sys/param.h 中的 HZ 常量决定,该值决定了系统的时间精度,在 Solaris 中的定义:

#define HZ              ((clock_t)_sysconf(_SC_CLK_TCK))
_SC_CLK_TCK 表示每秒的 clock ticks 数,也称为嘀嗒数
       由于所有 UNIX 系统都按照 epoch 来计算时间,对于 32bit 的 UNIX 系统时间范围为 1901.9042 ~ 2038.0958,这可能造成系统或者应用程序的问题,参考 eygle 的文章:
Oracle与Linux/Unix下的时间处理
Oracle诊断案例-Job任务停止执行


时区
       TZ 环境变量保存当前时区,在安装系统过程会要求设置一个默认时区。/usr/lib/zoneinfo 目录和其子目录中保存了所有支持的时区信息(Solaris 时区信息保存在 /usr/share/lib/zoneinfo 中)。TZ 环境变量的值必须包含在这些时区文件中,如果设置错误将会被重置为格林威治标准时间。
       在 Solaris 中可以通过修改 /etc/default/init 文件中的 TZ 值来修改默认的时区。为了兼容早期版本,可以为该文件建立一个字符连接:/etc/TIMEZONE。该文件的格式为每行一条 VAR=value,VAR 必须是 TZ, LANG, CMASK 和任何 LC_* 环境变量。
参考:http://bbs.chinaunix.net/viewthread.php?tid=877273

显示时间
       所有 UNIX 系统中都有 date 程序,用来显示系统时钟。date 程序会根据 TZ 环境变量显示合适的时间。

oracle DBALNP01 > echo $TZ
GB-Eire
oracle DBALNP01 > date
Wed Jun 13 07:54:38 BST 2007
oracle DBALNP01 > TZ=PRC
oracle DBALNP01 > echo $TZ
PRC
oracle DBALNP01 > date
Wed Jun 13 14:55:04 CST 2007
       此外,可以指定 date 的显示格式,语法:
date + format
       其中 format 必须是一个词,所以对于包含空白的 format 参数需要用引号包起来
oracle DBALNP01 > date +%y%m%d
070613
format 参数可以参考:
http://www.heysky.net/digest/2007/04/date_format.html
06:30 Some useful Windows Terminals commands (1954 Bytes) » Hey!! Sky!

       All colleagues must log into the Windows terminals to do their jobs in my new corporation.  There are only several  servers and all colleagues work on them, so sometimes we will be kicked out of the server. It always makes us depressed. And sometimes when I re-log into the server, I can't start Lotus Notes. Because the application is running, though I have been kicked out.

       I have encountered this kind of situation several times, so I want to find a way to resolve it. With google I find several useful windows command which I did not familiarize.

msg.exe: Send a message to a user local or remote. This is command is used frequently.
       msg username [/SERVER:servername] message
       Send message to username on servername

qprocess.exe: Display information about processes local or remote = query process
       qprocess username [/SERVER:servername]
       Display all processes beloned to username on servername

query.exe: Query TERMSERVER user process and sessions

quser.exe: Display information about a user logged on = query user

tsdiscon.exe: Disconnect a user from a terminal session
       TSDISCON [sessionid | sessionname] [/SERVER:servername] [/V]

tskill.exe: Kill a Terminal server process
       TSKILL processid | processname [/SERVER:servername] [/ID:sessionid | /A] [/V]

logoff.exe: Log user off
       LOGOFF [sessionname | sessionid] [/SERVER:servername] [/V]

       All the commands have an optional  parameter: /SERVER:servername. It is useful to operate other server.

06:30 SQL*PLUS 格式化输出 (2319 Bytes) » Hey!! Sky!

       需要写一个简单的脚本,将所有数据文件、控制文件、在线日志文件等输出到一个文件中,一口气写完之后,发现输出结果中每行之间都有一空行,非常不爽,研究了一下 sqlplus 的设置参数之后解决了问题,把几个常用的参数在这里做一个小结,方便以后查询。

set echo on/off                     是否显示脚本中的需要执行的命令
set feedback on/off               是否显示 select 结果之后返回多少行的提示
set linesize n                          设置一行最多显示多少字符,之前就是因为 n 设得过大,导致行与行之间有空白行
set termout on/off                 在执行脚本时是否在屏幕上输出结果,如果 spool 到文件可以将其关闭
set heading on/off                 是否显示查询结果的列名,如果设置为 off,将用空白行代替,如果要去除该空白行,可以用 set pagesize 0
set pagesize n                        设置每页的行数,将 n 设为 0 可以不显示所有 headings, page breaks, titles, the initial blank line, and other formatting information
set  trimspool on/off               在 spool 到文件时是否去除输出结果中行末尾的空白字符,之前的隔行可以用该参数去掉,和该参数对应的是 trimout,后者用于屏幕输出
set trimout on/off                   是否去掉屏幕上输出结果行末尾的空白字符
06:30 My readings released (2394 Bytes) » Hey!! Sky!

       虽然已经使用抓虾订阅了一些自己喜欢的站点,但一直想在自己的站点上做个 RSS 聚合器,只是自己太懒一直没搞。这两天抽了点时间终于把它搞了起来:http://www.heysky.net/readings/。后台程序用的是 Lilina,这是个开源程序,操作很简单,功能也不多,但挺实用,对于个人站点来说已经足够,很多个人站点都使用这个开源程序,比如最出名的车东,以及 eyglefennganysql 等。但由于原版的 Lilina bug 太多,而且动态输出在 dreamhost 上速度太慢,所以我使用了 anysql 的 hack 版本。该版本能生成静态缓存,再结合 wget 定时刷新缓存,现在使用起来效果不错。

       Lilina 安装挺简单,配置好 conf.php 文件,由于要对 .***.data 文件 和 cache 目录写入,对他们授予适当的权限,然后就可以使用 edit.php 添加 rss 了。我按照 CNOUG.net 的样子修改了导航条,然后是 css 的修改,以让界面和我的站点协调。对 css 还是不太熟悉,这里的调整花费了大量的时间。现在基本完工,还有几个问题需要以后再修改:

1.时区问题:Lilina 没有提供修改时区的功能,现在输出的时间都不对,我看了下 CNOUG.net 也是如此,暂时不管这个问题了。
2.图标问题:经过 anysql 的修改,dreamhost 上已经可以显示图标,但是没有图标的站点会显示一个空白,不太好看,想先用一个默认图标代替它。
3.无法添加 rss:Lilina 似乎对 rss 格式要求非常严格,一些不太符合规范的 rss 怎么也加不进去,暂时只能放弃了。
4.界面还需再调整。

       再花时间去修改也不知道是什么时候的事情了,先将就着用吧,哈哈,看上去还不错。

06:30 Getting a Traffic Ticket (1516 Bytes) » Hey!! Sky!

       In the afternoon, I added several english learning RSS to ZHUAXIA . One of them is ESLPOD.com(English as a Second Language Podcast). It is an interesting English learning website which have lots of short dialogues about different kinds of subjects. In every dialogue, firstly, two persons talk with each other with slow speed. And then another person explain the dialogue(words, phrases and usages). At last they repeat the dialogue with normal speed. It is useful to practise listening.

       Just now,  I listened to the ESL Podcast 267 – Getting a Traffic Ticket . Following is some notes.

Traffic Ticket 交通罚单
siren 警报声
pull over 开到路边
driver’s license, registration, and proof of insurance 驾驶执照,行驶证,保险证明
speed 超速
You were going 70 in a 55 mile an hour zone 你在限速55mph的区域以70mph的速度行驶
run a red light 闯红灯
brake light 刹车灯
tags 年检标签(贴在车窗上的那种)
to cite you 给你开罚单,citation 罚单
get away with just a warning 逃脱处罚,只给一个警告
06:30 Flickr 也挂了 (1876 Bytes) » Hey!! Sky!

       几天前才看到 Flavian 说 my.opera.com 被和谐了, 没想到才过没多久,噩耗又一次传来,这次轮到 Flickr 了。虽然所有的页面还都能访问,可是图片全部显示不出来了,这样和整个站点挂掉也没什么区别了。根据 William的分析,是 farm1.static.flickr.com 和farm2.static.flickr.com 关键字被过滤了,这招实在太狠了,不管是好图片还是“不好”的图片统统被和谐了。

       曾经以为网络是唯一自由的空间,现在看来自由早已不复存在。

=============
2007.06.13 update
=============

       听说 Flickr 发布了多国语言,其中包括繁体中文。上去一看,果然文字全部变成繁体中文了,应该语言自动判断的,但总觉得繁体中文看上去很拥挤,又改成英文了。

       William 说 farm2.static.flickr.com 域名已经可以访问了,我上去一看发现自己的照片都是在 farm1.static.flickr.com 域名上的,比较郁闷。不过在 Kamus 的博客上看到他在推荐一个 FF 插件,可以绕开 GFW,访问 flickr,试了一下果然好用,装上之后不用任何设置就可以查看 flickr 的图片了。

06:30 Differences Between Rowid & Primary Key Materialized Views (21094 Bytes) » Hey!! Sky!

       物化视图有两种不同的刷新方式,其中的 FAST REFRESH 对于数据仓库型的数据库相当于有用,它只将上一次刷新之后修改的数据刷新到本地。

       要使用 FAST REFRESH 必须在 MASTER 表上建立物化视图日志,用于记录对 MASTER 表的修改。Oracle 用两种方式来定位被修改的行,一种是 ROWID,一种是 Primary Key。在 Oracle 8 之前,只有 ROWID 这种方式被支持,从 Oracle 8 开始 Primary Key 方式开始被支持,并成为了默认方式。而为了向前兼容,ROWID 的方式也被保留。下面看看两种不同类型物化视图在创建过程中都做了些什么,有什么不同。

基于 ROWID 的物化视图

SQL> create table skytest(a number primary key);

Table created.

SQL> create materialized view log on skytest with rowid;

Materialized view log created.

SQL> select object_name,object_type from user_objects
  2  where object_name like '%SKYTEST%';

OBJECT_NAME                    OBJECT_TYPE
------------------------------ ------------------
MLOG$_SKYTEST                  TABLE
SKYTEST                        TABLE

SQL> create materialized view MV_SKYTEST
  2  REFRESH FAST
  3  as
  4  select * from skytest;
select * from skytest
              *
ERROR at line 4:
ORA-23415: snapshot log for "ORACLE"."SKYTEST" does not record the primary key

Oracle 8 开始默认使用 Primary Key 方式。

SQL> create materialized view MV_SKYTEST
  2  REFRESH FAST
  3  with rowid
  4  as
  5  select * from skytest;

Materialized view created.

SQL> select object_name,object_type from user_objects
  2  where object_name like '%SKYTEST%';

OBJECT_NAME                    OBJECT_TYPE
------------------------------ ------------------
I_SNAP$_MV_SKYTEST             INDEX
MLOG$_SKYTEST                  TABLE
MV_SKYTEST                     TABLE
MV_SKYTEST                     UNDEFINED
SKYTEST                        TABLE

        可以看到除了多了物化视图 MV_SKYTEST 之外,还多了一个索引 I_SNAP$_MV_SKYTEST 和一个表 MLOG$_SKYTEST

SQL> desc MLOG$_SKYTEST
 Name                                        Null?    Type
 ------------------------------------------- -------- -------------------
 M_ROW$$                                              VARCHAR2(255)
 SNAPTIME$$                                           DATE
 DMLTYPE$$                                            VARCHAR2(1)
 OLD_NEW$$                                            VARCHAR2(1)
 CHANGE_VECTOR$$                                      RAW(255)

       MLOG$_SKYTEST 表中记录了主表上修改的记录,M_ROW$$ 保存主表上修改行的 ROWID,这样可以定位修改的行。

SQL> select INDEX_NAME,INDEX_TYPE,TABLE_NAME,UNIQUENESS,STATUS,GENERATED from dba_indexes
  2  where index_name='I_SNAP$_MV_SKYTEST';

INDEX_NAME                     INDEX_TYPE                  TABLE_NAME                     UNIQUENES STATUS   G
------------------------------ --------------------------- ------------------------------ --------- -------- -
I_SNAP$_MV_SKYTEST             NORMAL                      MV_SKYTEST                     UNIQUE    VALID    N

SQL> select table_name,index_name,column_name from dba_ind_columns
  2  where index_name='I_SNAP$_MV_SKYTEST';

TABLE_NAME                     INDEX_NAME                     COLUMN_NAME
------------------------------ ------------------------------ ------------------------------
MV_SKYTEST                     I_SNAP$_MV_SKYTEST             M_ROW$$

SQL> select table_name,column_name from dba_tab_columns
  2  where table_name='MV_SKYTEST';

TABLE_NAME                     COLUMN_NAME
------------------------------ ------------------------------
MV_SKYTEST                     A

       I_SNAP$_MV_SKYTEST 是建立在物化视图 MV_SKYTEST 中 M_ROW$$ 隐藏列上的唯一索引,该列正是保存了主表上对应行的 ROWID 值,建立该索引应该是为了提高刷新的性能。

       另外一个类型为 UNDEFINED,名字和物化视图一样的东东不知道是干吗的。


基于 Primary Key 的物化视图

SQL> drop materialized view MV_SKYTEST;

Materialized view dropped.

SQL> drop materialized view log on SKYTEST;

Materialized view log dropped.

SQL> select object_name,object_type from user_objects
  2  where object_name like '%SKYTEST%';

OBJECT_NAME                    OBJECT_TYPE
------------------------------ ------------------
SKYTEST                        TABLE

SQL> create materialized view log on SKYTEST;

Materialized view log created.

SQL> select object_name,object_type from user_objects
  2  where object_name like '%SKYTEST%';

OBJECT_NAME                    OBJECT_TYPE
------------------------------ ------------------
MLOG$_SKYTEST                  TABLE
RUPD$_SKYTEST                  TABLE
SKYTEST                        TABLE

SQL> create materialized view MV_SKYTEST
  2  REFRESH FAST
  3  as
  4  select * from skytest;

Materialized view created.

SQL> select object_name,object_type from user_objects
  2  where object_name like '%SKYTEST%';

OBJECT_NAME                    OBJECT_TYPE
------------------------------ ------------------
MLOG$_SKYTEST                  TABLE
MV_SKYTEST                     TABLE
MV_SKYTEST                     UNDEFINED
RUPD$_SKYTEST                  TABLE
SKYTEST                        TABLE

SQL> desc MLOG$_SKYTEST
 Name                                                                                Null?    Type
 ----------------------------------------------------------------------------------- -------- --------------------------------------------------------
 A                                                                                            NUMBER
 SNAPTIME$$                                                                                   DATE
 DMLTYPE$$                                                                                    VARCHAR2(1)
 OLD_NEW$$                                                                                    VARCHAR2(1)
 CHANGE_VECTOR$$                                                                              RAW(255)

SQL> desc RUPD$_SKYTEST
 Name                                                                                Null?    Type
 ----------------------------------------------------------------------------------- -------- --------------------------------------------------------
 A                                                                                            NUMBER
 DMLTYPE$$                                                                                    VARCHAR2(1)
 SNAPID                                                                                       NUMBER(38)
 CHANGE_VECTOR$$                                                                              RAW(255)

       此时 mlog 中保存了 Primary Key。RUPD$_ 开头的表用于可更新的基于 Primary Key 的物化视图。

SQL> select INDEX_NAME,INDEX_TYPE,TABLE_NAME,UNIQUENESS,STATUS,GENERATED from dba_indexes
  2  where table_name in ('SKYTEST','MV_SKYTEST'); 

INDEX_NAME                     INDEX_TYPE                  TABLE_NAME                     UNIQUENES STATUS   G
------------------------------ --------------------------- ------------------------------ --------- -------- -
SYS_C001221                    NORMAL                      SKYTEST                        UNIQUE    VALID    Y
SYS_C001224                    NORMAL                      MV_SKYTEST                     UNIQUE    VALID    Y

SQL> select CONSTRAINT_NAME,CONSTRAINT_TYPE,DEFERRABLE,DEFERRED,GENERATED from dba_constraints
  2  where table_name in ('SKYTEST','MV_SKYTEST');

CONSTRAINT_NAME                C DEFERRABLE     DEFERRED  GENERATED
------------------------------ - -------------- --------- --------------
SYS_C001221                    P NOT DEFERRABLE IMMEDIATE GENERATED NAME
SYS_C001224                    P NOT DEFERRABLE IMMEDIATE GENERATED NAME

       同时,Oracle 也会在物化视图上建立相应的 Primary Key。不过对此我比较疑惑,由于刷新机制的缘故,Oracle 是不建议在物化视图上使用 Primary/Unique constraint 的,而建议使用 non unique index + deferred constraint 的形式,参见:Note:284101.1

       最后要说明的是,在创建物化视图日志时可以同时指定 WITH ROWID 和 WITH PRIMARY KEY。此时,在物化视图日志中两者都会保存,但最终物化视图是哪种类型的则是在创建物化视图时决定(只能指定 WITH ROWID 和 WITH PRIMARY KEY 其中一个)。如果所有的物化视图都是同一种类型的,那么建议在创建物化视图时也只指定那一种类型,可以减少空间消耗提高刷新性能。

参考:

Oracle materialized view mlog$ table

Differences Between Rowid & Primary Key Materialized Views
Note:254593.1

Fast Refresh Causing ORA-1/ORA-2291/ORA-2292, Complete Refresh Works Fine
Note:284101.1

06:30 转帖: 揭开美职场华人遮羞布 Chinese to be brave and compassionate 2017 (54203 Bytes) » 木匠 Creative and Flexible
注:放在一起,方便阅读。

美国华人三块遮羞布

最近不停的在网上各种群里看到抱怨“阿三”和“烙印”的声音,终于忍不住了。似乎美国职场华人的路,是被印度人堵死的,自己一点责任也没有。真的是这样吗?我把话说狠点,我倒认为,认为自己"怀才不遇"的华人,大部分其实是不愿意面对现实,而是胆怯的躲在了三块遮羞布的后面。

这三块遮羞布是

1. ”因为我们英文不够好,所以吃亏”

2. “因为印度人不好,会拍马屁,又抢了我的功劳”

3. “因为职场的华裔(专题)先辈不好,没有传帮带”

看出来没,反正都不是自己不好。都不怪自己,而是怪英文,怪“烙印”,怪华裔的前辈不给力。

英文不好这块遮羞布

是真的英文不好,还是因为言之无物,索然寡味?从我的观察,西方社会其实还是蛮公平,说话没人听,原因往往是下面几个:

没有内容

没有见地

没有逻辑

没有激情

没人在乎听你说什么的时候,最自然的反应,就是躲在“我英文不够好”这块遮羞布的后面,因为这样至少还保护了自尊。我倒是看惯了结结巴巴的人,因为说话有内容,有深度,有逻辑,有激情,大家就愿意去听。

建议看一下这篇文章,来想想是真的英文不好,还是交流能力不够?找到真正的原因,才能对症下药!

壳牌面试官八年记(1)| 什么叫交流能力(见文末附文)

印度人不好这块遮羞布

职场华人为什么常常被印度人超过?我的观察是,华人工作用功,而印度人工作走心。

印度人总在琢磨公司的“大事”。举个简单的例子比如公司组织架构,公司的远景和战略,他们常常比华人更加清楚。

好多华人员工,忙忙碌碌,连停下来和人说话的时间都没有。你不和老板交流,怎么知道你做的是老板需要的?你说印度人拍马屁,讨老板喜欢,可是人家其实是走心,时时刻刻换位思考,想着老板需要什么,这不比闭门造车强吗?

打个比方,我就见过中国同事,没理解老板想要造一个轮船的目标,啃哧啃哧造出了一个非常先进的飞机发动机。这老板能下得来台,能给他的老板交差吗?

公司大了,做事虽然重要,知道为啥做其实往往更重要,因为每个人都是颗螺丝钉,光自己好不行,还得是那个大机器所需要的。印度人这一点就常常很强:想老板所想,急老板所急,此为走心。

我看到好多美国或欧洲或印度的同事,每天花很多时间在和人说话,吃饭,喝咖啡。实在事情做不完,就下班后做。华人上班时间,总体而言,工作太多,说话太少,虽然用功,但没给人走心的感觉,似乎还是或多或少混口饭吃,一不如意就跳槽,所谓“此处不留爷,自有留爷处”。但跳来跳去,还是金字塔的低端。

最近参加了一个创业群,一位华人创业者聊起他在美国的经历。他说美国归根结底还是讲究和认可价值。在这个地方,只要你的产品或服务真的有价值,在这个成熟的市场,还是会有人认可。如果你觉得美国不公平,那你试试给我找到更公平的地方。

华裔前辈不给力这块遮羞布

这块遮羞布用的也很多。可是人家为什么一定要提携你呢?

我想起自己一件惨痛的教训。2006年我想换一个工作。当时一位华裔前辈帮我推荐,得到了他的公司面试。不是他的部门,所以他并没有亲自参与。我面试通过了,可同时也面试了两家别的公司,结果我没有接受他公司的聘用。这本身其实没什么,但我竟然都没有想起来和他沟通这件事,连通知一下也都没有,他后来知道了非常不高兴。

职场的互动,包括前辈后辈之间的交流,从来都不是单行道。前辈的提携,一定是有理由的,或者看到你的潜力,或者是你的思考或经历对他有些启发,否则人家的激情来自哪里?

结论

今天说的极端了一点,也刻薄了一点,其实就为了扩大一点效果。其实我知道,在美国的华裔,很多很多真的非常优秀,各方面都很优秀。从加州到波士顿(专题),从纽约(专题)到休斯顿(专题),各行各业我都亲眼见到很多很牛逼的华人。所以网上的那些论调,其实我觉得非常可悲。每个人要为自己的行为和结果负责,而不要把责任推脱给别人。

还记得我刚进公司的时候,一次打算和部门总裁为一件小事吵架。一位小伙伴说的话,我至今记忆犹新:

“Even if it's justified, if it's not constructive, then don't talk or complain about it !”

(“就算你要说的或者抱怨的事,理由是充分的,如果它不是建设性的,那就不要去提!”)

对“烙印”的抱怨,我觉得就坚决属于此类。就算你觉得理由再充分,事实再确凿,也于事无补。不如多花点时间来提高我们自己的实力,而不要像祥林嫂一样成天去抱怨印度人。

附:壳牌面试官八年记(1)| 什么叫交流能力



我从2008年开始参与壳牌的招聘,到现在已经8年。最初的4年主要是在美国做斯坦福大学的校园招聘,近4年则是在中国做电话面试和壳牌招聘日。这些都是我正式工作以外所做的义工,虽然花的时间不多,但一路走来,也觉得收获颇多。

今天是端午节,是高考后第一天,而六七月份也是大学毕业生面临职业选择的关键时刻。回想起自己以前刚毕业的时候,心中充满困惑。因此我决定开始写一些心得。我不会写具体的壳牌的招聘程序,而是在不泄露公司机密的大背景下,分享我的一些应该说是放之四海而皆准的经验。

首先纠正一个错觉。很多人可能都以为,面试官的工作,就是给应聘者出难题,如果答不上来,就被淘汰了。我不是这么认为。一个真正优秀的公司,它的面试过程应该有两个目的:一是宣讲自己公司的价值观和优秀之处,让应聘者被吸引;另外一个则是主动的去发掘应聘者的优秀之处,如果可以发掘到足够多适合自己公司所需的优秀素质,而又没有足够让人担忧的瑕疵,那么就可以被聘用。所以作为面试官,我有的时候会很格外的努力,去帮助应聘者在面试的过程中去反思自己的素质,尤其是对那些不擅长在较短时间内迅速地展示自己长处的应聘者。

还记得有一次参加面试官的技能培训时,教官说了一句我至今记忆犹新的话:

“大家也许能意识到,(面试的)那几个小时,通常都会是 TA 们人生中最忐忑的时刻之一。”

正是因为这一点,在我试图提高自己面试技能的过程中,我一直坚信的一点就是,一个真正优秀的面试官,最需要的就是能善于发现应聘者的独特的优点和长处。

今天第一篇随笔,谈谈交流能力。再说明一下,以下纯属个人观点,与壳牌完全无关。

1. 什么叫 communication?

根据岗位的不同,一个面试官去考察的应聘者的主要素质也会不同。很多公司都会考察交流能力,即所谓“communications".

在我看到的和听到的关于交流能力的所有表达中,我最喜欢的既简单又深刻,扣中要害的是:

“交流能力是两个圈:一个圈是能够把自己的意思表达清楚;第二个圈是在表达过程中,能考虑对方的反应和感受。一个优秀交流能力的人,是处于两个圈的交集。”

(1)能够把自己的意思表达清楚

也许很多人会说,作为成年人,难道我们不会说话吗?事实上一个简单的问题,表达清楚并不是很困难。可是在职场,很多时候我们需要对复杂问题做一个完整表达,这其实并不容易。我曾经写过一篇比较长的文章,专门讲这个议题。今天只重复一个故事。

这个故事是关于我的一次旅行。那时我初进职场,在一家公司做工程师。一次我和一个朋友,以及他的几个朋友,去科罗拉多滑雪。他的朋友中好几个人都是大牛。比如有一个是贝克休斯亚洲区总裁,还有一个是一家非常优秀的咨询公司的创始人,名叫 Ellen。

有一天滑雪完,我们泡温泉聊天。期间我问了 Ellen, 我说我自己觉得英文不错,表达也不错,可为什么老板年底评估的时候,给我的反馈说我需要提高表达交流能力。

她说有没有可能是你表达方式的问题。我们在职场的任何一种表达,都需要一个好的框架,而这个框架中,最重要的就是它的逻辑性:

-问题的根本和来源是什么?

-条件和假设是什么?

-有哪些可供选择的方案?(“发散”式思维)

-最后建议的方案是什么?(“收敛”式思维)

-被选中的方案将来会在什么样的条件下可能有什么样的变化?

这就是一套标准的阐述复杂问题的流程。如果你缺了任何一环,那就或者不完整,或者不具有说服力。

而所提方案和建议,必须“先放后收”,即有一个完整的从“发散”式思维过度到“收敛”式思维的过程:

-先”放”是为了思维的完整性,证明你将各种可能性都考虑到了;

-后“收”才有行动力,因为最后必须去选择一个方案去实施和执行。

现在很流行麦肯锡的思维方法。书店里就有很多他们的书,比如金字塔方法,比如不漏不重的原则(MECE - mutually exclusive, collectively exhaustive), 等等。Ellen 所说的“发散”式思维的过程,其实也就是 MECE 原则。

(2)在表达过程中,能考虑对方的反应和感受

第一,是要简明扼要,抓住重点。适当有趣有帮助,但不是必须。

职场里很多人,可能都有体会,如果一场会议有一个参与者,说很多虽然正确但没用的废话(所谓的 "motherhood statement"), 时间长了大家都会很累。这种表达方式,就是忽略了接收方的反应和感受。

还有很多人,尤其是职场新手,很可能有一种倾向,就是想把自己所有的工作和辛苦,向上司展示出来。这样的做法,是对上司珍惜时间的需求的忽略,和对上司耐心的极大考验。曾经作为职场菜鸟,我得到的最好的一个来自上层的建议就是:

“Don't tell me the pain, tell me the results”(不要告诉我你工作的辛劳,告诉我结果)

公司的经理,常常处于信息缺乏的状态。他们最需要的下属,就是可以把信息用严谨的思维(比如不漏不重原则)来过滤和解释,并指出盲点,形成建议,这样才可以做出基于信息的决策。

第二,是要Positive / constructive, 即正面和有建设性。

谁喜欢一个总是抱怨和很消极的的人呢?就算你的表达是对的,接受方的反应和感受也最多是不情愿的接受,而更可能是反驳。

我的一个同入职场的朋友说过一句话很好

“Even if it's justified, if it's not constructive, then don't say it."(“就算你的(意见或抱怨)是有理有据的,如果它不是建设性的,那么就不要说”)

这样的例子太多了,但都有点敏感性,就不举了。有心人可以自己留心一下,不出三天,一定可以在身边找到例子(欢迎留言分享)

第三,是要有良好的互动

要对事不对人,有一个 vigorous debate (积极的讨论)。不要 defensive (过于维持自己的观点而不接受别人的反馈)。

我另一个职业导师 (mentor) 曾说过一句话,我也体会很深,那就是 “Feedback is a gift” (反馈是一个礼物)。所以对别人的反馈,至少在起始时,要 “always assume good intentions" (永远先假设别人是好意),否则就把可以接受反馈的大门全关上了。

良性互动还有一个关键是要双赢, 而不是老想赢了别人。(Win win,not to win)。很多职场菜鸟,尤其是名校毕业生,老想证明自己比别人聪明,这样常常反作用。"one can show that he or she is smart, but not smarter (than others)"(一个人可以通过行动证明自己的聪明,但不要竭力去证明自己别人更聪明)。在职场比如会议中,如果很明显的让别人显得愚蠢,往往并没有积极作用。

最终,如果在互动中,你成为别人积极职业体验的一部分,那你的表达和交流就是成功的。

PS: 我的建议

  任何人都应该试着写一些文字。

  我在读博期间,参加了美孚实习生的面试。正规面试之后,在闲聊中,面试官得知我一直有一个 blog (类似微博)。后来入职后我才明白,这件事竟然帮助了我,因为他认为,喜欢写文章的人,一定会持续不断地有意识的去努力提升自己的表达能力。


PS: 最后,一个家里刚刚发生的小故事...

  五岁的儿子在玩沙坑,然后对他妈妈说,“你拍一下”。

  她以为是要把沙拍平一点,就动手了。儿子很沮丧,说我是要你拍照片!

  回家路上,他很不高兴,说“妈妈我说的话你怎么总听不懂啊!” 沮丧的要哭。

  几个小时以后,老婆向他道歉,他很成熟的说,“妈妈没事,我原谅你了。”

  之后又说:“我好想长大到6岁!等我六岁了,我要赚钱,买一种机器,让大人能听懂小孩的说话!”

  我笑的不能停。这真是记忆好深好有趣的一刻。 
06:30 招聘方会问到哪些问题? preparing 5 interview questions (1855 Bytes) » 木匠 Creative and Flexible
when you're preparing for your interview, make sure you've got examples to cover all of these bases.

The five questions are:
  1. What brings you to this interview? (Why you're looking for a new job and why you're interested in this company)
  2. What value will you add to our company? (How your skills will be directly applicable to the job)
  3. Can you work well with the team?
  4. What is special about you?
  5. What's your salary and when you can start?

收录备用。

06:30 干掉左逼,还我们一个干净的世界 erase extremists (1100 Bytes) » 木匠 Creative and Flexible
“我们作为父母,对学校给我们孩子进行的性别教育忍无可忍!我们拒绝对孩子进行图像化的、扭曲的、支持滥交的性教育! 维多利亚吸毒致死的孩子的父母的发言让人落泪。“不是每个嗑药的孩子都是坏孩子”;“希望他不被人记住是个因为这个而走的,而是记住他是个幽默的、爱阅读、爱运动的好Boy” 政府不让父母取得孩子的医疗记录,孩子可以自己做决定。过度讲人权。 extremlists 这个左逼变态政府用渐进渗透式推广吸毒,推广各种变态事物。 静坐这种求饶似的抗议方式,太被动。 我老乡说的对,我们不能被当软柿子捏,我们要主动出击,打倒这个左逼变态政府,再踩上一只臭脚,让他们永世不得翻身。 我们要学习美国很多州积极组织起来参政的华人那样,制定法律,禁止亚裔细分,惩罚哈佛大学。 我们也要积极参政从政,制定法律,禁止以后任何组织提及这些推广滥交,吸毒等等,乌烟瘴气的恶毒想法。
06:30 带车进美国 import car from Canada to USA (3523 Bytes) » 木匠 Creative and Flexible
今天介绍怎么带加拿大的汽车进入美国。工作和生活使用。

我记得处理过程很简单。

带上加拿大的买车发票和保险文档,证明你是车主。
到美国海关(CBP port, 最好是路过加美边境的海关),开具进口证明。免费。
  CBP entry form, Form 7501
  Link, https://help.cbp.gov/app/answers/detail/a_id/218/~/requirements-for-importing-a-vehicle-%2F-vehicle-parts
检查尾气排放符合标准。十年内的新车一般都符合标准。
stating that the vehicle conforms to EPA and DOT standards
  Link, https://www.epa.gov/importing-vehicles-and-engines/importing-canadian-vehicles

换美国驾照。我是华盛顿州,$85。- Get a WA driver license.
  华盛顿州不用考试。
  北卡州需要简单的考试,考察路牌图形的含义。
带上海关进口证明,到车管所(Department of Transportation),换Title和车牌.
  Title纸 证明你是车主。
  华盛顿州,$160.  http://www.dol.wa.gov/vehicleregistration/licenseplates.html
  北卡州,$400 左右。

就这些了。 去之前,记得打电话确认,少跑冤枉路.

具体细节查询各自相应的网站。

我换车牌照和 Title 的办公室,下午4点以后去,不用等待,服务还行:
1175 NW Gilman Blvd B3, Issaquah, WA 98027
Alpine Licensing


Q & A

根据文中的链接,除了7501表格,还有EPA 3501-1和DOT HS-7表格也要填吧?关于尾气排放,我确定发动机有符合EPA的铭牌了,还要去检测否?我还未去换美国驾照,是否要先去,还是等换车牌一起去?

A: 不用提前填表。在海关一次办理,现场填.
A: 按照上面文章里面的顺序办理.
A: 发动机检测在海关完成。打开前盖看一眼。

一个表都不用填?7501呢?

A:不用填. 在海关现场填。非常简单。

过海关是否一定要工作日?

A:提前打电话确认。
我走的是弯路。没有在边境海关办理.
我是在西雅图南边一个海关办公室办理的。
那里的警察比边境的警察和蔼。
(海关地址:6 South Nevada St, Seattle, WA, 98134)

也就是说我周日回的话,办不了,索性不办,可以找别的海关办

A:可以. 官网推荐在边境办理,节省时间。

06:30 小石头走路队之麦当劳山 little stone team hiking mount McDonald (3962 Bytes) » 木匠 Creative and Flexible
本次走路,计划不周,准备不充分; 但是临时应变处理是相当的成功,致使我们在弹尽粮绝,手机电尽,无导航的情况下,下午4点赶回来出发点,远远好于预计的天黑之前.

早上一觉醒来, 9:20am, 煮饺子是来不及了. 忽然想起昨天去唐朝老乡家拿的两个肉夹馍还在冰箱,烤烤匆匆吃完,9:50am.拉上邻居湖南小帅哥,急急忙忙赶往出发地点.

参加走路的人空前的少,一共三个壮汉. 穿过黄金溪露营地,顺着平坦的废弃铁路前进.
走着走着,发现不妙,水淹铁路,被迫走钢丝一般的走铁轨,由于紧张,数次踩进水坑. 过了水淹那段铁路,大家都能轻松走单轨. 感觉跟打高尔夫球一个道理,放松, Chillax.

然后就是trail导航软件把我们带到了铁丝网围墙,寻找消失的山路,开始第一次临危应变.
我们没有原路返回,而是继续前进,到达公路交叉处,有一个人(我),走到停车场,开车过来接大家,来到第二个登山出发点: Mount Wells 停车场.
而且我们明白了带叉灰线是铁丝网.

导航地图有一段像 trail 的线路,原来是 flowline , 输水管道. 只好沿着公路走向 trail 入口,狗绳也忘了带,用背包做成简易狗绳,勉强把朱丽叶牵到安全区域.

麦当劳山是走路队数年以来碰到的难度最大山路,其陡峭程度超出了 Mount Finlayson, 虽然路程短很多. 然后是果然有绳索,还是双保险,我们都在由衷的感谢修建绳索的志愿者.
中间朱丽叶上演了一段小插曲,在第二段绳索的起点,朱丽叶没能爬上去,悬在半空,着急委屈的呜呜哭了起来,意思像是说我怎么比人差呢? 从来都是我遥遥领先的. 然后就垮的一下摔了下来,一直退到第一段绳索的最低端,来回慢慢踱步,仿佛在思考什么,突然启动,一口气冲破所有障碍,顺利越过. 不由得赞叹朱丽叶跌倒爬起的勇气.
手机充电不足,错过了许多精彩的照片,下回来补上.

由于早期挫折,有点担惊受怕,原本打算原路返回. 可是又想着为下次走路做准备,三个男子翻过山头,去打探另一面下山的路.
导航软件指示可以绕圈回到停车场,4公里平路,当即决定不走回头路. 此处深山老林,少有人烟,一只黑熊从我们面前游荡蹦跳而过,大伙都在议论称赞黑熊的爬山本领,好身手.
走到一半处,有警告牌出现,同时导航地图显示有灰色叉线,完了,又有铁丝网围墙挡路,难道我们必须原路返回,此时虽未筋疲力尽,但我们已经丧失了再走一遍的勇气.
开始紧急应变,决定走过去看看,到底有没有围墙,地图上连续出现两道障碍,也没能阻挡我们冒险的勇气,终于顺利通过,长出一口气.
最最后,被水库的围墙拦住,有好心山民提示我们可以绕围墙而入. 在望着我们满脸疲惫,他还问我们想走一个小时,还是半个小时,跟我们幽默.

轻轻松松回到了停车场. 下来就是等着因为穿越禁区导致的罚单了, 路上有人提起人脸智能识别技术,再一想,我们节俭的政府才不会费劲搞这些高科技.

我们一共走了12公里,略逊于 华盛顿山 走路支队今天的17公里,该支队分流我们大批人马.
年轻队员说我们也完成了一次速度与激情.

好了,下回可以带领大家走一段精彩的环线.



06:30 如何在工作生活中 高效管理时间 ~ GTD + Workflowy + Tomato (4705 Bytes) » 木匠 Creative and Flexible
各位IT同仁好,

鄙人迟钝, GoLang 还没有修炼到可以分享的程度.

如果大伙不嫌弃, 九月份, 鄙人可以跟大伙分享一下我5年的 GTD 使用经验, 如何使用 Workflowy 和 西红柿, 在工作和生活当中 高效的管理时间.


Fyi,

鄙人的 twitter 的一些摘抄,



任务多,忙起来 以后,仅仅使用GTD是不够的。一定要采用 GTD 加 西红柿工作法,两者互相配合,威力暴增,工作效率和产出率陡然上升。


来了项目三个月,终于开始干活了。任务多,忙起来以后,GTD方法就开始显示威力了。作为GTD初中级用户,着重DRY,每天 Review 任务优先级,善用 waiting box 和 trash box.


Thanks,
Charlie 木匠 | Database Architect Developer

06:30 加拿大公民的美国工作税务问题 Canadian work in USA pays tax (537 Bytes) » 木匠 Creative and Flexible
=
问:加拿大公民在美国工作,可以不用给加拿大交税吗?

答:如果满足以下任一条件,就必须交。


  • 有房子等不动产在加拿大。
  • 有直系亲属(配偶和孩子)在加拿大生活工作。


如果都不满足,还要检查其他一些细节。


----------

根据我个人经验,加拿大税务负担更重。加拿大需要补税的额度是美国交税总额的 50%. 还真不少.

06:30 你想找什么样的工作呢? What computer job do you like ? (5232 Bytes) » 木匠 Creative and Flexible
鉴于目前飞速变化的技术趋势,你想找什么样的 IT 工作呢?
置身于什么样的主流技术栈 可以让你的技能树相对长期的维持在市场需求的浪潮巅峰?


谢提问。 一直在思考这个问题, 但是没有深入想。
我没有美国绿卡。 用的是TN状态。
目前是美国的第三个工作, developer manager, 全凭运气得到的工作。


目前有两个方向:


一个就是继续目前的道路,慢慢往上爬, Director, VP ...


一个就是像 Liang 学习,做软件架构师, Liang 是我的榜样。
需要苦练 JavaScript , Golang 这些基础语言的基本功,在一线战壕摸爬滚打三五年。



您是怎么思考这个问题呢? 欢迎踊跃评论和指点。



下一篇会有精彩续集。


谢谢,
木匠.

06:30 你想住在哪里呢?Where is your favourite living place? (2769 Bytes) » 木匠 Creative and Flexible
你想住在哪里呢?

维多利亚只有两个季节,半年夏天和半年冬雨天。

经历十二个漫长到阴雨湿冷季节以后,咱想去温暖的南方,晒晒身上的湿气。

基本筛选条件是:气候好,治安好,经济发达。
南美洲除了古巴,治安都很差。
非洲经济落后。
第一批找到三个候选地:澳大利亚,加州 圣地亚哥 和 夏威夷。

随后被朋友们否决了澳大利亚。
原因是 ... ...。
圣地亚哥也在被迫接收大批穆斯林难民,治安每况愈下。

所以现在剩下到候选地只有夏威夷了。

昨天在朋友家聚会,新生一个方案,不必彻底搬家。
每年冬天12月和一月能够在夏威夷 或者 古巴,远程工作两个月。
生活就美好许多了。

咱现在目标就是,投资自己,修炼提高,然后可以找到许多远程工作。


根据 Hacker News Hiring Trends 排名,
前三位技术是:React, Python, JavaScript.
我决定猛攻 React + JavaScript.

Fyi, here is the Trending Developer Skills.
Top 3 are ReactPython and JavaScript.

https://medium.freecodecamp.org/trending-developer-skills-based-on-my-analysis-of-ask-hn-whos-hiring-26c02a3ca1fd

06:30 为啥 Amazon 的 DBA 比较累 worn out (6940 Bytes) » 木匠 Creative and Flexible
Q: A quick question, are Amazon DBA jobs good? I remembered you said you know them very well.

A: 我认识的亚马逊的DBA们,都走完了。

那里把人都累死球了。  在西雅图臭名昭著,员工待遇 排名倒数第一。

Q: 各位老同事,晚上/ 早上好,为啥amazon的DBA比较累,要维护的数据库系统实例多还是每个实例压力大容易出问题?还是别的原因?谢谢。

A: 李大哥兴趣这么大,鄙人就简单回答一下。

首先回顾一下去年春天,我在北京的可怜样。 人生无常,抓紧时间享乐。哈哈哈。

为什么亚马逊臭名昭著呢?

末位淘汰制。
Raising the bar. 把人往死路上逼。
许多领导向上负责,希望出成绩,向下逼迫,不知道他代表的是谁。
每周工作60到80小时。
客户第一,员工末尾,向投资人负责。 所以可以放心的购买亚马逊股票。

“要维护的数据库系统实例多还是每个实例压力大容易出问题”,
这些是次要的,却也让人受不了,一个人平均维护50个 production 数据库。
全是小学生文化水平就可以干的重复劳动,体力活。
SDE(软件工程师)设计开发的低性能数据库,全靠DBA们擦屁股。权利和责任脱节。

一个云计算AWS部门的北京小姑娘,一晚上被叫起来20次,忍无可忍,离职去了CostCo,
倍感生活舒畅,身体发福,生了两个胖小子。

现如今,从印度抓来大批廉价劳动力。公司里面90%是老印。

好了, 最好祝各位健康幸福快乐! 有空来西雅图和维多利亚玩耍。

06:30 一封家书,关于邪恶的美帝国. American fear The middle kingdom (1758 Bytes) » 木匠 Creative and Flexible
老爸的家书:

    今天读报,又一个叫许家强的华人工程师遭美国“钓鱼执法”,记得早前天津大学一教授如法遭难。北京一经常处理中美商业诉讼案件的律师董纯钢说,起诉中国人,首先对于美国来说是“政治正确”的,由上至下各个部门也都很支持。因为从民事上告,可以从中国人身上拿到钱,从刑事上告,又可以限制中方的产业结构升级和转移。二三十年前,日本、韩国公司也曾遇到过这种情况。他提醒说,既然无法回避这样的事实,那么中国一些从业者就更应洁身自好,以免陷入诉讼之争。
    我大段引述报上所载,是因为你现在美国打工,联想到安全为要。我过去常想一件事,就是美国是西方国家之最,民主、自由、博爱、平等是他们的价值观,国内法律齐备,这些在本国以内可畅行其道。如若搬到世界上,在国与国之间,那就反其道了。这就是矛盾,大大的矛盾!我也意识到,国与国之间与人与人之间的关系的相处是一个道理。我最近看到比较权威的一家报纸说,国际间国家都是势利国,如同人都是势利眼一般。大概我的意识不太离谱啦。

我的回复:

完全正确。美帝国主义是邪恶的。
坚持独立思考和批判性思维。

目前就业环境还是美国最好,但同时我们要保持冷静和时刻警惕。革命尚未成功,任重而道远。

还好美国有了川普( Donald Trump ),虽然违背“政治正确”,但却会带来实实在在的好结果。因此推荐华人选民投他一票。
06:30 一些扯淡的职业中介 debunk job agent (2744 Bytes) » 木匠 Creative and Flexible
鄙人:I talked to some friends work at RTP, they said these Java jobs salary is not attractive, no one is interested  ^O^. Is there some bonus and stocks besides base salary?

猎头: There are bonuses offered with these base salaries.  While they might not be the highest paying job opportunities they offer working with Big Data (Hadoop) and other skills that could propel someone moving forward.  We also have other Sr. Developer positions.

鄙人一针见血: I don't think the employer just want to train the employees and then let them go, to get a better compensation job. Ha~ha.

猎头自己打圆场: Haha understood Charlie.  Thanks for checking with your network though.

06:30 一个 Development Manager 的自我修养 (1435 Bytes) » 木匠 Creative and Flexible
采访两个 Software Developer Manager, 做了一些笔记,供给日后学习和提升.


" Lead, but not manage. "


部门协调

Clearly set objectives and expectations.
取舍。 do, cut, trade off.
or put to backlog, or late delivery.

Partner.
- Treat Product Manager/BA/QA/OPS as partner
- Set priority and plan.
- Think Win/Win

任务安排

Function spec,
Break down to use cases,

Star team member: architect design, key function development.
intermediate: common functions
Junior: simple duplicate work, support work, learn system.

Grow team

Make team member happy, then proactive and motivate from inside out.
- Service
- Enable

Teach or set the goal and measurable outcome.
 ( No micro management.  )
- Review the progress and help on problems.
- Code review and learn.

- See the specialty strong point, and adapt to it.
-- Talkative ,做沟通协调的活
-- Quiet,给出具体目标,定期检查进度,尽量少打扰。

Performance Review
- monthly review task completion progress, lunch break talk,
- Reward outstanding key members
- Encourage the rest.

Hiring
- what do we want to have? to fit into the team.
- passion
- looking for potential.
06:30 people skills on manager 搞好领导关系 (1287 Bytes) » 木匠 Creative and Flexible

  • 关心,真心愿意帮助(态度,发自内心)。
  • 目标和需求。理解并清楚明白领导心目中的整个团队的目标。
  • 紧密反馈:决策,过程和结果。
  • 言辞积极。

 非拍马屁式的搞好领导关系.
06:30 Where to go and what to invest 人员流向和价值投资 (3637 Bytes) » 木匠 Creative and Flexible
我们这一代人,有两个重要的东西,没法从父母身上学到。

一个是毕业以后去哪里。一个是价值投资。

这些都完全不能指责我们的父母,因为他们那个时代,这两件事情都是不存在的,他们毫无经验和心得。

毕业以后去哪里?为什么去经济发达的大城市?

1. 资源。
2. 包容 (文明程度高)。

展开说说资源,

比如说我,刚刚在维多利亚开始从事大数据工作,整个维多利亚只有两个大数据职位。
我如果失业,基本就没地方去了。
而大数据是兴起不久,欣欣向荣的朝阳职业。很容易在加拿大和美国大城市找到一份待遇丰厚和环境舒适的工作。

选择城市,其实和选择职业方向的道理是一样的。

我们大维多利亚地区,目前只有两个大数据职位。 JavaScript 前端职位数量是大数据职位40倍以上。
如果我考虑长期呆在维多利亚,就必须考虑转向前端工作,用户交互界面的开发。

在比如,如果你是从事自动驾驶设计的,或者橄榄球运动,那么你在维多利亚会饿死。 ^_^

这也吻合了 雷军雷布斯 的猪飞上天的理论。
我们必须顺势而为,而不能和大环境抗争,就像冲向风车的唐吉坷德。

再说说投资。

规划好的一生,投资回报是后半生主要经济来源。
怎样对抗通货膨胀,怎样做到资产保值和升值,主要依靠正确的长期价值投资。
可以参考我的另一篇文章:http://smallgoodwish.blogspot.ca/2017/11/2017-nov-investment-idea-and-decision.html

最简单的方法就是长期投资五百强指数或者道琼斯指数,这样你已经打败了 90% 的个人投资用户和大多数基金。

什么是长期,就是像房子一样长期持有,等你卖房子的时候,也不能抛售。
然后就是每月定期存入,比如每月存两千。

如果在中国,主要投资就是买房和供房。做好二十年长期奋战的心理准备。

06:30 USA telecom jobs 美国工作机会: data migration Project Manager (5331 Bytes) » 木匠 Creative and Flexible

继续上干货.  有兴趣的请留言. 或者发邮件: dba.myra  at gmail


Rate: $80+ / hour.
If you have any friends that has very strong Project Management data migration skills? Please take a look at the job description below and let me know:

Project Manager – BSS Transformation
Ericsson/T-Mobile
Position Name: Project Manager - BSS Transformation - Testing
Status: Open
We need a strong candidate, PM who has had significant data migration skills. Please send resume ASAP. Preference would be local Seattle.

Scope of Work: 
7+ years of experience in managing complex systems integration projects.
Strong technical background with proven track record of managing complex BSS systems.
PMP certificaiton preferred.
Experience in establishing project controls to ensure controlled progress, including TEST MANAGEMENT, reporting, tolerances and governance structures.
Deep knowledge of Project Management and Quality Management concepts and principles.
Sound knowledge of telecommunication technologies and trends.
Excellent skills in Test Management is required.
6 month renewable contract (this project has plans to go through December 2015)
Location: Washington-Bellevue
Target Assessed Level: Project Manager II
Weekly Pay Rate – discussed with candidate on the phone when recruiter receive’s resume.

06:30 To be a full stack developer (2035 Bytes) » 木匠 Creative and Flexible
Oracle 数据库市场日渐狭窄。 $60/小时 成了市场标准,而且每况愈下。 这个技能越来越不好卖。
雇主只顾捡便宜货,伯乐难求。

开吻说了,搞IT,要走在技术趋势的前面,才能长盛不衰。
开吻就是渐悟加顿悟,从困境中走出来的。以前在偏僻的小岛 长期用C++做一个安装工具,倍感冷清和落后。如今快速冲刺,成为全栈工程师的先驱,跑着硅谷的前面。
(先驱: 本义指马车中前面的一匹奔跑的马,引申指在思想上和行为上走在前端的人。)


那么什么是全栈工程师呢? 我个人喜欢下面这些解释:

a genuine interest in all software technology. (对所有的软件技术抱有一种真挚的兴趣)。

full stack developer (FSD) 其实没有什么硬性的门槛,需要的是解决任何问题的能力和意愿。你要做到的就是不固步自封在一个领域。遇到问题,就去研究,
不因为问题不在你的 comfort zone 就放弃或者推给别人。即使一开始的解决方案很笨拙也无所谓,just learn whatever it takes to make it work.
比如说我要做一个网站,我有一些东西没碰过,但我有足够的兴趣和动力去搞个八九不离十。(这里自学能力很重要,有好的mentor也会帮助很大)
当你经历过一次这个过程以后,你就会有信心去弄明白更复杂的东西,在之前的基础上进一步去消化、改进、学更多的东西。
学的东西用来解决或是改进实际遇到的问题,由实际问题驱动.

~ 作者:尤雨溪
~ 链接:https://www.zhihu.com/question/22420900/answer/21457250

他们学习技能和知识,不是为了成为某个领域的专家;而是因为那些 是完成自己目标所需要的.  ~顾鹏


~ 第一版草稿.  25-Feb-2016.
06:30 Replace car battery 给汽车换电池 Ford Edge (5725 Bytes) » 木匠 Creative and Flexible
有工具的话自己装就好了,简单的很。 
Using a combination wrench or a socket and ratchet。

一般的车可以直接拆掉电池的负极,然后拆下正极,电极连线顶端有个铁圆圈,往上提,就拆下来了。
电池底部侧下方的位置有一个固定电池的卡子,把卡子上面的螺丝卸开后直接把电池拆下来,拆卡子上面的螺丝需要一个带延长杆的扳手。

注意:先拆负极,先装正极。

参考:

ratchet 就是可以接延长杆的扳手。

这样一套工具在 O'Reilly Auto Parts 大概二十多块钱。 

我第一次操作,结果画蛇添足(Overengineered),把套正负电极的铁圈上面的螺丝也给拆掉,又得重新拧上,冬天站在户外,雨雪交加,不是很愉快。

往回套上铁圈的时候,用 ratchet 手柄敲击固定,结果拧螺丝的铁盖套子掉进前仓。
同事汽车专家说,不用担心,除了忍受咣当摇晃的噪音,不会造成损害,晃着晃着它自己就会掉下去。

汽车电池好重啊,要使点劲。

2017-01-05.

06:30 Looking for renting a room in Seattle/Bellevue (1344 Bytes) » 木匠 Creative and Flexible
Hi There,

How are you doing?

I'm looking for a room to rent in Seattle.
Hope it takes less than 30 minutes drive to Seattle downtown in rush hour, quiet and lot's of trees. 

E.g.: Mercer island, Factoria/Bellevue, Magnolia, Interbay, Ballard, Bainbridge Island.

Myself:
-=-
42 years Computer Database Engineer.
I just started a new job in Seattle downtown 3 months ago,

I'd like to have a quiet place, so I can read and learn every night.
love hiking, swimming and soccer.

My home is in Canada, I usually go back home every weekend.

Never drink or smoke.

My linkedin profile: 

AirBnb review,
=-=

Could I have your detail address? Can I have a look at your room?

My contact:  Good.GoodWish at gmail.com

Thanks,
Charlie

06:30 IT 职业的出路和方向 What computer job last longer regarding machine learning ? (18138 Bytes) » 木匠 Creative and Flexible
这篇博文先从一段对话开始。

Q:从你的博士专业学习来看,机器代替人写代码,还需要多久?
A:已经可以了

Q: 那么,还有哪些方向 暂时不会被机器人替代?
A: 心理咨询 

Q: 离我太远,有离我近一点的吗?比如:有哪些代码 机器人比人写的差?
A: 当manager.

Q:这么说来,中级十八湾的程序员机会,很快就会飘走了?  
 (link: http://zhu1.blogspot.com/2016/07/it-role-trending.html )
A:从大局说 决定做什么的职位很难自动化 执行怎么做的职位很快飘走.

技术角度来说 深度不如广度 .

十年java编程经验 不如十种语言各一年.

Oracle 五年经验 不如五种数据库各一年.

Q:开始在哪里找这样的机会入门呢?
A:各人性格经历不同吧。

Q: 自己创造机会?
A: 现在忙,回头慢慢说.


正文开始:

现在有空了 说说我的想法 抛砖引玉.

技术出身的,不想做纯程序员了有三个方向:管人,管设计,管进度.

管人是管理资源调配,简单说是要把好钢用在刀刃上。关键是要定义什么是好钢,什么是刀刃。资源包括钱,人员,时间。对手下的人要能看出来有没有能力(没能力的不招,有能力的培养),对平行的组要有交流沟通谈判的能力,对上级要有建立良好个人关系(拍马溜须)的能力。

管设计是管理技术生态系统,title往往是architect, team lead, specialist 之类的,这个要求对技术有广度的了解,对外会评估供应商产品的优劣(难点在于定义评估标准),对内评估解决方案的具体设计(如果现有的不好,要能说出来怎么改就好了),对上级要能说明白要钱要人来干什么(向无技术背景的人科普),对技术人员要建立信任关系,作为能解决技术难点的高大形象出现。

管进度是管理输入和产出效益。软件行业的生产资料是人,而每个人不一样。有个笑话说一个女人生孩子要十个月,认为十个女人生孩子要一个月的是项目经理。管进度的人最大的挑战是identify 关键路径。什么可以并行操作,什么必须先后顺序执行,谁可以有效接替谁的工作,在关键路径的人忙得不亦乐乎的时候,非关键路径的人应该做什么(为下一步作准备?培训新技能?)管进度的人将定义整个团队的最大产出。这个产出是团队相关(每个人不一样),产品相关(每个需求不一样),时间相关(第一次做和第二次不一样)。

在亚麻(AMZ)里,管人力资源的是manager,管产品资源的是product manager, 管设计的是architect, 管进度的是TPM. 各个公司title不一定相同,文化不一定相同,所以肯定job description 有出入。不过基本分这几类.

作为纯技术人员,从十万到二十万容易,再往上基本要走leadership 路线。从技能角度,要学习communication communication communication. 怎么建立信任,怎么建立个人关系,怎么交流,谈判,劝说,鼓励,培养,怎么很客气的说不,怎么很礼貌的骂人,怎么很优雅的说草泥马你丫白痴你老子怎么没把你射墙上。

说完管理路线,再说说技术路线。很多技术人员宁可跟技术打交道也不愿跟人打交道,因为技术更可靠,输入输出对应更合理更可预期。这是个人性格决定的,没有好不好,只有适不适合。技术路线最大的好处是从一家公司到另一家公司几乎完全transferable. 这家三年Java经验,几乎跟另一家三年Java经验水平完全一样,所以换工作几乎没有经验损失。但这个transferable 硬币的另一面是几乎完全replaceable. 这个有三年经验的人换成另一个有三年经验的人公司也几乎没有技术损失。因为replaceable 所以工资不会太高,也有被新技术取代的风险。比如软件测试工程师曾经占到软件从业人员的一半,现在越来越被自动测试取代,这个职业已经像电话接线员一样要慢慢退出历史舞台了。

管理职位因为有大量的投入在relationship, 所以从一家公司换到另一家损失很大,公司和个人都损失很大,所以管理层往往比技术人员更稳定。同样的,管理层找新工作难,找到同样工资的新工作更难,因为很多东西带不到下家。难的另一个原因是公司的金字塔结构决定管理职位要比技术职位少得多。因此如果走管理路线,要有在一家公司多待几年的打算。

全文完。


咋样?有启发吧?  ~ 木匠

06:30 IT role trending 软件行业的职位收入趋势 (4097 Bytes) » 木匠 Creative and Flexible
一直洋洋得意觉得自己收入不错。和一群一起寄宿的小年轻聊天,顺道广告我们公司的中级Java developer 空缺,年薪 100k~110k。
当他们获知这样的工资,没有额外福利奖金期权;马上一脸茫然,表示毫无兴趣。

其中一个小哥说了一席话,感觉自己太失落。初级十三万,中级十八万,高级三十万。他刚刚从硅谷搬到西雅图微软,数据来自他的朋友圈,大多在西雅图一流软件公司。听完后 我备受打击。

- 消息准确吗? 

以前20年,总认为自己走在前面。突然发现自己落后太多,备受打击。 是绝望,还是希望? 赶紧修正方向。


附录,一些朋友的反馈:

坛主:人和人不能比,下次聚餐告诉你一些真实的人生故事。比这工资差距大得多的多。走自己的路,enjoy 你自己的life.


长枪会长:对 人不能跟别人比 只能跟自己比 有的人选择事业 有的人选择家庭 有的人选择悠闲 只要自己尽力了就好 。
你走遍五湖四海 也有多少人羡慕呢 包括我。
我被lay off 的时候才觉得自己失败呢 后来不也挺过来了 很多事要经历之后才明白。
生活不仅是眼前的苟且 还有未来很多年的苟且。

06:30 IT 2016 novice, 1. 对同胞善良。 2. 对老印团结。 3. 对白人实际. (6133 Bytes) » 木匠 Creative and Flexible
分享一篇很实在的文章,给还不是职业老油条的朋友一点参考.

其中三点我深表赞同,并且身体力行。

1. 对同胞善良。
2. 对老印团结。
3. 对白人实际.  白人泛指加拿大和美国土生的天生说英语人群。实际是少情感,多冷逻辑。

===========
发信人: eversprint (征程),
标  题: 好公司好工作的特征是善良和成长
BBS 未名空间站 (Sat Apr  9 21:59:16 2016, 美东)

为了家庭团聚,我在Boston坚持当了好几年码工,现在领导毕业,总算可以奔向西海岸
了。总结起来在Boston的码工经历就是“后悔”,非常后悔!
因为这里的IT公司基本上既不善良,也无成长。

说说我自己和周围朋友的经历,来说明为何这么评价。

1. 何为善良:
某软件公司是由外国移民创立,曾经是花街的宠儿,后来毕竟过气,只要业绩不好就裁
员。可能因为主力是金砖国家移民吧,H1b生效就开始办绿卡,对被裁人员采取月底离
职,月中15号就让人回家,剩下15天对绿卡申请处于I140 pending的补办加急PP。 下
岗员工拿着package和I140走人。

2. 何为不善良:
某公司是世界五百强,产品线也有软件一块。对技术人员一律要求工作两年以后才开始
办绿卡。倒霉的人不到两年或者两年多就被迫离职。有人好不容易熬过两年开办绿卡的
,赶上经济危机来了,又停办,两年后再重新开始,折腾到第七年才交140.和四大会计
所一个待遇了。如果是单职工的话,中间几年自求多福吧。

好的IT公司应该是入职以后尽早开始解决身份问题,猥琐地琢磨着用身份卡人的,只能
说这是个烂坑!

某公司是这里仅有的一两家牛互联网公司,也要求至少工作一年以后才可以办绿卡,
RSU少得可怜。

这里的大型IT公司用人的基本原则就是把码工当制造业或者是会计业的雇工来使唤,没
有用股票补充收入的Partner文化。就算是优待购买股票,不是给补贴2000元就是来个
95折,呵呵。当然了西海岸的不少IT公司也是啥都没有的。

3. 何为成长:
某小公司,Full stack 文化,前端 后端 middle ware一起来,在这里半年时间工作学
到的东西比大公司当螺丝钉好几年学到的都多。

缺点就是没有“培训”这种福利, 一刻也不闲着,没有安全感, 有合同工的感觉。

4. 何为不成长: 不少老牌软件公司用的Tech Stack老旧极了,离开了以后如果没混个
绿卡或者起码弄个PD,基本上是浪费生命,因为技术价值清零。问题是这么老旧的东西
人家还要挑三拣四的。

我后来到这里的一家互联网公司面试的时候对HM开玩笑说我就问三个问题: 用GIT吗?
有免费Lunch吗?full stack吗?

把简历里面那些过气的东西删除吧,他们都是负资产。
如果呆在那些技术老旧的岗位,考虑离职吧,职业生涯的前方大概率不是绿洲,而是悬
崖。

5. 何为浪费: 某著名老牌传统软件公司, 居然骄傲到了如此程度:QA招的都是工程
专业的PH.D, 反正Boston的PhD多的是, 一个人力资源的买方市场,这事挺悲哀的。

总结起来,一个好的工作机会就是要么善良,能最快或者最大可能地办绿卡,要么就是
成长,把你往死里用,但是你很快从初段上升到中高段。

最好的机会就是两种好处都有,有一个就是很好了,如果既不善良,又无成长,请麻溜
溜地滚蛋,否则将来后悔的是自己,机会成本的概念有哇?

最后,说点自己的建议:

1. 对同胞善良。 这个国家作为少数派,只有自己人能理解你,帮助你。我以前有个先
于我入职几年的中国同事甚至提防我,希望Layoff的时候能幸存下来,呵呵,这点出息
。有机会还是帮帮同胞吧,能力是你自己的,不会因为别人而失去竞争力。

2. 对老印团结。这个东方国家的很多文化和中国是类似的,比如集体主义,吃苦耐劳
。他们和我们一样是来异国打拼的。想想在美国,Linkedin的network里面,除了中国
人外,到底是老印愿意和你交朋友,还是白人和你愿意来往? “统一战线”这个策略
也适用于美国。 多学习人家的优点吧。老印普遍愿意去读MBA,他们在美国发达的秘密
就在这里,我不想展开说这个了,有心人自然明白。

3. 对白人实际。基督教白人社会的基本原则就是“自私”是被鼓励的。我强烈建议对
东亚圈的朋友保持与人为善的东方文化,对资本主义社会长大的白人采取“该乍样就咋
样”的做法。不要心存幻想, 要见国人说人话,见白鬼子说鬼话。

4. 对自己拼搏。不管是转专业的,还是科班生,保持终身学习的心态吧。美国给了更
多享受物质文化的机会,但是也是一个远远比中国残酷的社会,玩物就会丧志。别看人
家如何爽,人家的绿卡是配偶或者爹妈搞定的,你也有这个运气吗?美国强大的原因是
自由资本主义推动的敢死队文化,赢者嚣张,输者活该。

5. 人尽其才。 不管别人如何评价你,尽量去做自己喜欢的东西吧。刷题可以涨自信,
学技巧,但是真正的系统是一行行的代码堆出来的,No magic. 而且,有些东西没那么
难,缺乏的是脚踏实地的去做。把名校毕业生的小聪明收起来,天资不能自动变成产品
, 中间是苦役。
06:30 Canada TN visa 加拿大公民如何办理去美国工作的 TN 工作许可 (4301 Bytes) » 木匠 Creative and Flexible
以下内容,普遍适应于IT相关职位,根据鄙人2014年自己办理TN的经验。
(如果雇主请律师,就更简单了)

主要是 offer letter, 和一封 TN petition letter。一般雇主都会提供。
注:Offer letter可以不用,雇主都会请律师写一个TN Letter,里面有你的简历,工作内容和待遇。 如果用 Offer letter,必须写在公司信签纸上。

需要注意雇佣期(小于3年),职位,薪水等细节,保持前后一致。
如果有本科学历,职位可以不用 Computer System Analyst,
任何一种工程师都比 Computer System Analyst 容易通过。比如:软件工程师,电信工程师.

其次是学历证明 和 工作经验证明。

如果不是在加拿大和美国读的大学,需要学历认证。
学历认证: http://www.foreigndegrees.com ,$82,大概一两周就可以办好。
(BC省本地只能找 BC IT,又慢又贵,无需自找麻烦)
如果在中国读的大学,需要大学的成绩单,毕业证原件。
- 各种材料证明,最好都能有原件。

可以找前雇主出具工作经验证明,匹配 TN 网站的职位描述,但是不能照抄.
三年以上相关工作经验比较好。
http://en.wikipedia.org/wiki/TN_status#Recognized_TN_professionals


SSN:以后慢慢补充。我是2001年 去美国的时候办的。

海关:听说飞机场的TN审查是最难的,尽量去人少的边境口岸。
提前准备好如何叙述自己的工作内容,吻合TN描述的专业内容,而不能照抄。

拿到TN以后,必须立刻肉身前往美国。飞机,轮船,汽车,自行车,步行 都可以。
因此会要求看你的飞机票,船票等等。

如果经常走陆路来往于美国和加拿大,可以顺便把nexus办好,以后出入海关方便。

Q and A.

Q: (学历认证)是要寄去原件,对方认证后原件随认证寄回来吗?
A: scan copy 就行。

Update:

鄙人2015年六月八日在(维多利亚)悉尼海关顺利办好第二个TN。
美国边境警察大姐一见面,就让我通过了。
然后一边吃着早餐(快中午十二点了),一边办理各种琐事手续,大约花了20分钟,包括我去售票处取现金,换零钱。
建议是少说话,非问不答。见到这么和蔼的签证官,我不免多聊了几句:比如警察工作辛苦,一天十二小时,没时间吃饭。表达人文的关心。

期间最难的两个问题是:

  • 我们只收现金,50元整。
  • 美国的固定住址。(可以用公司地址)

跟第一次TN相比,真是天上地下,那一次边境警察领队非要让我来来回回叙述工作内容。
就怕加拿大人过去抢了美国人的工作机会。

附录:

这里是需要的材料清单:供参考。



2015-June-03
木匠

出门打高尔夫球去了.

06:30 Better than t-mobile (1462 Bytes) » 木匠 Creative and Flexible
For last 2 years, I only know T-Mobile is affordable. Now we have some better choice.

If you decided to go to Ultra Mobile, I strongly suggest you to go EZKonnect and get either 6 month or 12 month.  If you get 6 month, it’s like $21/month.  If you get 12 month, it’s like $19/month.

If you pay month by month, it’s $29 plus they charge some $1 and some other charge every month, the total is $31/month.  If you pay 6 month or 12 month, they only charge that $1 once.  It’s quite a bit saving.

Also just for your information, I bought a moto e phone (spring locked) from Bestbuy for $29.99 + tax, and got the Walter Hagen plan from https://ringplus.net/.  It’s unlimited call, text and 1G LTE data for free.  You only need to top-up $20 one time and make sure not exceeding that data limit.  I got it for my daughter.


Enjoy,

Charlie 木匠.
06:30 2017 个人总结 sum up (2221 Bytes) » 木匠 Creative and Flexible
2017年,是我这辈子职业发展最差的一年。

2017年一年内失业两次,这辈子首次发生。

第一次失业,激发了我果断改变职业方向,放弃 Oracle 传统关系数据库,转向就业率最高的 JavaScript React 前端应用开发。
第二次失业,加上机缘和一些朋友的分析建议,我转向到职业生命周期更加耐久的大数据。

2017年,也是我这辈子职业发展最好的一年。

一年内找到两份工作,两次成功就业。

第一次,还没有学完 freecodecamp,没有拿到培训资格证书。就通过面试 开始上班了。
要感谢我以前积累的数据建模和系统结构设计知识。
工作中,完整的设计和开发了一套 React 前端网页应用。
接触到了大部分目前流行的 React 前端工具和技术栈。
比如:Redux, Apollo GraphQL, Glamorous, ES2005

第二次,只准备了一周 Spark 基础入门和简单的 Python 编程。通过严格的4人车轮面试。
要感谢我以前深厚的 SQL Analytic Function 功力。充分练习各种 Behavior question 的应答。
更要感谢我积攒的人际关系,在以前的共事过程中赢得了老同事的充分信赖。
工作中,主要应用目前大数据最前沿的技术,Scala, Spark, Kafka,  AWS EMR, S3 and Hive, plus Python, PostgresQL and Tableau.

最后,引用雷军和刘德华座谈语录,作为总结:
在哪里跌倒,千万不要在那里爬起来,自己慢慢爬到风口,再爬起来。- 顺势而为。


附录:赠送:

IT技术风向标:https://www.hntrends.com/  帮助你找到大风口,和猪一起飞起来!



06:30 You can't trust everything you see... (2543 Bytes) » The Tom Kyte Blog
A few years ago, I wrote a book - Effective Oracle by Design.  Oracle Press really wanted my photo on the book and I really didn't want it.  So they went around me and asked Oracle Magazine for the head shot they had and were using for the print column back then.  Unbeknownst to Oracle Press - that head shot was a photoshop job.  It was my head, someone else's body.  I never did find out whose body it was - but it was definitely not me.  If you have the book and look really close - you can see the line on my neck where they slid my head on.

I just got the latest issue of Oracle Magazine and lo and behold - they've done it again.  My column has a new picture of me on the intro page, one that I've never seen before - but only because it never happened in real lift.  Again - it is my head from a shot I recognize, but on someone else's body - not just neck and shoulders this time - but the full body!



This time - I know where the body came from.  In an effort to make people as unproductive as possible - I'm going to offer up a challenge.  If you can point me to the source of the body shot on the internet, I'll send you a copy of Expert Oracle Database Architecture - 3rd edition when it comes out (which will be a while, sometime in 2013 - I'm not starting the edits on it until the next release of the database is official).

In order to play - supply a link to the source body here in the comments.  Make sure to use an identifiable username (do not use anonymous) or identify yourself in some way in the comment.  If you are the first to get it right - I'll post a comment myself stating so and ask you to email me so I can get your contact information (so you'll have to check back every now and then).  When the book is printed, I'll send you a signed copy. table 42 :))))))
06:30 When is a foreign key not a foreign key... (5716 Bytes) » The Tom Kyte Blog
I learn or relearn something new every day about Oracle.  Just about every day really!

Last week I was in Belgrade Serbia delivering a seminar and an attendee reminded me of something I knew once but had totally forgotten about.  It had to do with foreign keys and the dreaded NULL value.

Many of you might think the following to be not possible, we'll start with the tables:


ops$tkyte%ORA11GR2> create table p
  2  ( x int,
  3    y int,
  4    z int,
  5    constraint p_pk primary key(x,y)
  6  )
  7  /
Table created.


ops$tkyte%ORA11GR2> create table c
  2  ( x int,
  3    y int,
  4    z int,
  5    constraint c_fk_p foreign key (x,y) references p(x,y)
  6  )
  7  /
Table created.

Looks like a normal parent child relationship - a row may exist in C if and only if a parent row exists in P.  If that is true - then how can this happen:

ops$tkyte%ORA11GR2> select count( x||y ) from p;

COUNT(X||Y)
-----------
          0

ops$tkyte%ORA11GR2> select count( x||y ) from c;

COUNT(X||Y)
-----------
          1

There are zero records in P - none.  There is at least one record in C and that record has a non-null foreign key.  What is happening?

It has to do with NULLs and foreign keys and the default "MATCH NONE" rule in place.  If your foreign key allows NULLs and your foreign key is a composite key - then you must be careful of the condition where by only SOME of the foreign key attributes are not null.  For example - to achieve the above magic, I inserted:

ops$tkyte%ORA11GR2> insert into c values ( 1, null, 0 );
1 row created.

The database cannot validate a foreign key when it is partially null.  In order to enforce the "MATCH FULL" option of a foreign key - you would want to add a constraint to your table:

ops$tkyte%ORA11GR2> alter table c add constraint check_nullness
  2  check ( ( x is not null and y is not null ) or
  3          ( x is null and y is null ) )
  4  /
Table altered.

That will ensure either:
  • All of the columns are NULL in the foreign key
  • None of the columns are NULL in the foreign key
As long as that constraint is in place - your foreign key will work as  you probably think it should work.


06:30 What I learned new about Total Recall... (699 Bytes) » The Tom Kyte Blog
Today I learned something new about the Oracle Total Recall option from a good friend (they were very happy to have told me something about Oracle I didn't know :) ) - it doesn't exist anymore!  The Total Recall option has been made part of the Advanced Compression option!!

Check this out http://docs.oracle.com/cd/E11882_01/license.112/e10594/options.htm#CJACCDBA to see what you get with the Advanced Compression option which now includes the Total Recall option as well - the Flashback Data Archive capability.
:)))))))))))
06:30 Upcoming Events... (5109 Bytes) » The Tom Kyte Blog
I took some time off from the road at the beginning of 2014 - getting ready to get back on the road again, lots of trips scheduled from February till the end of June.  This is just looking at public events for the next three months, hope to see you at one of them!

Virtual Developer Day Feb 4th

I'll be speaking, along with Jonathan Lewis, about the last 20 years or so of technology and where we think it is all going as part of the Virtual Developer Day - an entirely online event with many technical speakers.  Something to definitely look into!

RMOUG Feb 6th-7th

At the beginning of February, I'll be at RMOUG Feb 6th and 7th.  Truth be told, this won't be a trip for me - I'm located in Denver, CO now (since the beginning of the year).  I will just be returning from Toronto at the beginning of the week (I know all of the best places to go in February!).

Ohio Feb 11th-13th

Of course, if it is February - it must be Ohio.  That is the best place to fly into and then drive around in during that month :)  I'll be coming down from Ottawa (another 'must see' location in February) and spending Feb 11th in Dayton, Feb 12th in Cincinnati and the 13th in Columbus.  Hitting all of the user groups across the state.

Real World Performance Netherlands, Germany and Bulgaria Feb 18th-21st

Then I'm off to continue the Real World Performance tour with Andrew Holdsworth and Graham Wood.  This month we are hitting the town of Breda in the Netherlands on the 17th, Munich in Germany on the 18th and finally Sofia in Bulgria on the 21st.

Hotsos Symposium Mar 3-6th

If it is March, then it is definitely time for the annual Hotsos Symposium.  I'll be returning this year with a few sessions.  It should be nice and warm in Dallas in March!

Ireland March 11th-12th

I'll be in Ireland on March 11th for the Ireland OUG conference.  I'll be speaking on performance and the new In-Memory capabilities coming in Oracle Database 12.1.0.2 soon.  On the 12th - I'll be delivering a one day Optimizer Master Class.

Real World Performance London and Latvia

The tour continues with a date of March 26th in London and March 28th in Riga Latvia.  

Belgrade Serbia, April 1st-2nd

I'll post more details when I have them, but I'll be doing a two day seminar on the optimizer and developer related topics at this time.  It will be in conjunction with Oracle University so you can monitor their event website for information soon too.

IOUG Collaborate April 7th-11th

I'll be doing a one day optimizer seminar and four other sessions during the week.  April in Las Vegas - perfect time to be there!  And Admin Savage will be speaking too - love the mythbusters!

Oracle Conference on the James, April 24th

I'll be doing a keynote and a technical session in the morning of the 24th for the VOUG and HROUG.  Lots of great speakers here - definitely check this out if you are in the Virginia area!

Manila, Philippines - week of April 28th

I'll be doing some events in Manila this week - details to follow soon!
06:30 UKOUG 2012... (2432 Bytes) » The Tom Kyte Blog
The call for papers for the UKOUG 2012 conference ends in less than three short weeks!  If you were planning on going to the conference (and even if not) - you should consider submitting a paper.

I've been a long time supporter of all of the user groups and their conferences and I can attest to the quality of the UKOUG event.  The conference is chock full of technical talks with hundreds of sessions to choose from.  There is something for everyone there.

If you've never presented before, don't let that deter you from submitting a paper.  No one knows the anxiety that public speaking can bring better than I - I've written about it before. You'll find the conference to be an entirely different experience on the other side of the podium.  In addition to the experience of presenting, the networking and exposure that comes with being a speaker won't hurt you at all.  Whether you are a DBA or developer - having good public speaking skills is a necessity today - and using the conference as a way to build those skills is a great way to start.

Additionally - what you have to say is important and relevant to the user community as a whole.  A good conference needs a lot of speakers, from many diverse disciplines, with diverse backgrounds - the more speakers the merrier.  Don't think you don't have anything to offer - everyone does.  And don't feel that your topic wouldn't be interesting to someone else - it will be.  There are a lot of people out there trying to do some of the same things you've done and they'd love to hear how you did it.

That is one of the things about user groups I really like - they bring together a lot of people doing similar things - but in a different way.  You'll learn something new - and they will too.  

The UKOUG is one of the larger and well run conferences out there - don't be afraid to talk.  Challenge yourself to get up there and just do it.  You won't be sorry (ok, maybe in the minutes leading up to it you will be - but you'll get over that :) ) 

Hope to see you there - and don't chicken out!
06:30 The keys to Oracle… (4605 Bytes) » The Tom Kyte Blog


This is a question I get on asktom frequently – what are the things I need to know, what do I have to do to become expert, where is the list of key things I need to do with regards to Oracle.  It is a hard sounding question that has an easy answer.

If you are still wet behind the ears – or even just damp – then the definitive place to start is the Oracle Server Concepts Guide. Not only is this free - but it is your guide to understanding how the database works.  If you understand something - you can use that thing.  If you don't, well, simply put - you won't be able to effectively use it.  Getting a solid knowledge of how a transaction is done, what locking and concurrency controls are and how they work, how the database makes the data durable on disk (redo, undo management) - all of it is key to effectively using the database.

If you don't know who can see what data, which version of data, at what points in time - confusion will abound and data integrity will be lost.  Getting a good basic knowledge of how the database manages data is the only way for you to know how to code a correct program.

Once you've mastered the material in that document - I would suggest taking a look at the 2-Day guides.  They are a good way to get spun up on the necessary knowledge for either a Developer or DBA track. There is the 2-Day Developers Guide and the 2-Day DBA Guide.  These documents are designed to get you going - you won't be an expert after reading them, but you'll know the gist of what it is you need to know, to learn.  You'll have a good idea of what is available feature wise at the very least.

After these 2-Day guides (you can read either one or both), you might want to move onto some of the other overview guides - the 2-Day Performance would be a must read for anyone (followed by the Performance Guide itself eventually).

While all of this is going on - you should also be communicating - a lot.  Constantly.  Get on the forums - http://otn.oracle.com/forums, start writing - start asking questions (after reading the guidelines!!), start participating.  It is the way I learned much of what I know now myself.  But communicate - a lot.  You won't be able to progress forward in a vacuum.  Share everything you have in your head - and find others that will share back with you.  You'll find you not only get to learn (and to teach eventually) from people all over the world - but you may end up calling some of them your really good friends.

And then add time, a lot of time.  Maybe years of time.  You will not become really good at what you are doing in six months, not in a year, probably not in five years.  It will take a while - you need to have lots of different experiences, encounter many different situations, attempt and fail to solve many problems - before you'll become really good at what you do.  You might feel like you are expert in a year - but trust me - you aren't.  And four years later you'll realize how little you knew and how much more you have to learn.

I learn something new everyday about Oracle, just the other day something about Total Recall but many things in general, technical and otherwise - and I've been doing Oracle things for a long time.
:)))))))))))))
06:30 Sponsored: 64% off Code Black Drone with HD Camera (58 Bytes) » Delicious/Fenng/oracle
Our #1 Best-Selling Drone--Meet the Dark Night of the Sky!
06:30 Raw Devices... (338 Bytes) » The Tom Kyte Blog
A quick request...

If you are using RAW devices for your database (no ASM, no filesystem - just 'raw') - please drop me a line and let me know.  Include the size of  your database as well please.

thanks!
06:30 Oracle OpenWorld 2013, where I'll be... (2341 Bytes) » The Tom Kyte Blog
I have a fairly busy schedule next week at Oracle OpenWorld 2013.

It all starts on Sunday with a day long seminar.  I'll be doing a series of "five things about" - in the areas of SQL, PL/SQL, Performance and more.  You can check out and register for those sessions (and many others) here: http://www.oracle.com/openworld/oracle-university/index.html

Next, on Monday I'll be delivering:

Session ID: GEN8579
Session Title: General Session: What’s New in Oracle Database Application Development
Venue / Room: Marriott Marquis - Salon 8
Date and Time: 9/23/13, 12:15 - 13:15


I've done a session like this every year for the last 8 or so years, It'll cover what new Oracle Database Application development techniques have become available over the last year (and will include a few Oracle Database 12c repeats from last year where relevant).

Next, on Tuesday I'll be doing:

Session ID: CON8426
Session Title: The Five Best Things to Happen to SQL
Venue / Room: Moscone South - 103
Date and Time: 9/24/13, 10:30 - 11:30


This is not just the five best things to happen to SQL in Oracle Database 12c - but rather - five of the coolest things to happen to the SQL language over the years.  You might be surprised at how many you did not know about....

And lastly - on Wednesday - the last slot before the appreciation event - I'll be presenting on:

Session ID: CON11637
Session Title: What’s New in Oracle Database 12c
Venue / Room: Moscone South - 103
Date and Time: 9/25/13, 17:00 - 18:00


This is not a repeat of last years "Top 12 Things about Oracle Database 12c", this is all new content - things discovered and found useful in the last year of using Oracle Database 12c.  So, if you have seen my "12 things" talk - or videos - or read the articles in Oracle Magazine, this will be all new and different.

I hope to see you around - I'll be on site all week from Saturday evening until Wednesday night (flying back home on a redeye Wednesday).

Enjoy the show!
06:30 Latin America OTN Tour.... (1854 Bytes) » The Tom Kyte Blog
Just starting the Latin American OTN tour for the week here in Uruguay.

This is my first time in Uruguay and I spent the weekend in Montevideo.  It was a wet, damp weekend - very foggy too!  But on the plus side - the food is excellent and plentiful.  The local user group leaders here made sure we were well fed and warm for lunch every day :)

Today is the OTN conference - Dimitri Gielis is presenting right now using a "slide free" presentation.  He is building an APEX application to the specification of the audience, in real time.  A pretty neat concept, I like it, very good way to show the ability of the tool:

2012-08-06_14-57-22_202

Tomorrow Andrew Holdsworth, Graham Wood and I are doing a Real World Performance day here before we are off to Argentina for a repeat.  To get to Argentina, we are going to a three hour ferry ride.  It is supposed to rain - so that will be a damp, wet trip for sure.

After Argentina we'll be off for Sao Paulo Brazil for a repeat on Saturday (yes, Saturday!).

If you are in the area, you can find the details here: http://www.oracle.com/technetwork/es/community/user-groups/otn-latinoamerica-tour-2012-1634120-esa.html  or on http://asktom.oracle.com.  Hope to see you here!

Now I'm off to answer some questions that I have in my queue...
:)))))))))))
06:30 In Israel... (691 Bytes) » The Tom Kyte Blog
I'm in Israel now getting ready for the http://www.iloug.org.il/ conference in Jerusalem next week.  Looks to be a good one with Mark Rittman, Joel Kallman and many others speaking!

See http://www.iloug.org.il/TechDays/?page=BigData_BI#track  for the agenda...

The weather is beyond phenomenal here - see https://twitter.com/OracleAskTom/status/353171810784772098  for a quick picture.
06:30 East Coast Oracle Users Group Conference... (739 Bytes) » The Tom Kyte Blog
I'll be speaking at the East Coast Oracle Users Group Conference held in North Carolina on October 16th-18th.  It is always nice to be able to stay in my own timezone for a change!

There are opportunities to not only attend the conference but also to speak - they are still accepting abstracts for presentations until June 21st.  If you have done anything interesting (you have), please consider speaking out about it.  Others will be interested.  You never know how it will go unless you try!

Hope to see you there!
06:30 Doctors... (4746 Bytes) » The Tom Kyte Blog
This recent blog post by Seth Godin reminded me a lot of my introduction to Effective Oracle by Design of a few years ago. What was true then is still so true today...

Here is an excerpt from my book that mirrors what he just wrote:


I will use yet another analogy to describe how this book will present information.  Pretend for a moment that the developer is instead a medical doctor and the application is the patient.  There are many types of MD’s:

  • The emergency room (ER) doctor. They do “triage” – separating the hopeless from the ones that can be helped.  Performing quick fixes to keep patients alive for as long as possible.  They strive for short term band-aids to fix up the patient.  They will take a patient with a heart attack induced by smoking, bad diet and no exercise and get them stabilized.

  • The operating room (OR) doctor.  They get the patient after the ER doctor has triaged them and patched them up.  They strive for long term fixes to keep the patient not only alive but as fully functioning as possible.  They perform the by-pass operation on that heart attack attempting to clear the arteries.

  • The physical therapist (PT).  They get the patient after the operating room doctor is finished and begin a long and painful (not to mention expensive) process of rehabilitation.

  • The preventative medicine doctor.  They strive to avoid the above three doctors at all costs.  They counsel the patient to quit smoking, eat a healthy diet, and exercise – developing a phased plan to get them in shape.  If they do their job right – with the exception of unfortunate accidents (like a car accident), the patient will never see the ER, OR or PT doctors.
Now, the world needs all types of doctors – accidents do happen after all.  But one of the most important types of doctors is that last one, the preventative medicine doctor.  The one that tries hard to avoid having their patient need the other three.

It is my belief (experience) that most people and books approach tuning using the mindset of the first three doctor types above.  They are in support of the hero developer; e.g. the ER or OR doctor.  Perhaps that is partially due to my observation that pre-emptive good design and implementation is mostly a thankless exercise.  These developers seem to get all of the fame as they snatch the patient from the grasp of death (save the system by doing something miraculous).  They get called in at the last moment; work horribly hard for an extended period of time trying to keep the patient alive (and get paid handsomely as well).  The physical therapists are the unlucky souls that get the system after the ER/OR doctor has patched it up.  They are the ones responsible for keeping this system going.

I feel I am well equipped to speak from that perspective.  I am in fact one of those “heroes”.  I am called in to “lay hands on” systems and make them better.  I could write that book, I’ve been told I should write that book – but I won’t.

What is missing is the comprehensive approach that includes the preventative medicine doctor training.  There are some out there – my favorites being Guy Harrison’s developer book, as well as Jonathan Lewis’s DBA book.  These books, including my own “Expert One on One Oracle” work to remove the need for the hero.  Remember – firefighters are heroes when they do their job, we all just hope wenever need them personally!
06:30 Coming soon to Ireland and the Netherlands... (1207 Bytes) » The Tom Kyte Blog
In September I'll be delivering a free Seminar in Dublin Ireland from 9am till 2:30pm on September 19th at the Gibson Hotel.  It is being delivered in conjunction with the Ireland OUG and anyone (even non-members) may attend.  Food is involved so it is a real deal :)  I'll be talking about big data, statistics and managing large sets of data.  We'll close up with a Q&A session before lunch.  Full details on registering and the agenda may be found here.  Also see the Ireland Oracle User Group page.

After that, on September 20th, I'll be delivering a full day seminar with Oracle University in Utrecht in the Netherlands.  Full details - including the agenda - may be found here.

Hope to see you at one of the events!
:))))))
06:30 Checking out Adaptive Execution Plans in 12c (316 Bytes) » The Tom Kyte Blog
Just a short note pointing out a new video on how/when adaptive execution plans happen on Oracle Database 12c: http://www.youtube.com/watch?v=9o9iuxNBciQ&feature=youtu.be
06:30 All day optimizer event.... (1232 Bytes) » The Tom Kyte Blog
I've recently taken over some of the responsibilities of Maria Colgan (also known as the "optimizer lady") so she can move onto supporting our new In Memory Database features (note her new twitter handle for that: https://twitter.com/db_inmemory ).

To that end, I have two one day Optimizer classes scheduled this year (and more to follow next year!).  The first one will be Wednesday November 20th in Northern California.  You can find details for that here: http://www.nocoug.org/ .

The next one will be 5,500 miles (about 8,800 km) away in the UK - in Manchester.  That'll take place immediately following the UKOUG technical conference taking place the first week of December on December 5th.  You can see all of the details for that here: http://www.ukoug.org/events/tom-kyte-seminar-2013/

I know I'll be doing one in Belgrade early next year, probably the first week in April. Stay tuned for details on that and for more to come.
06:30 12c - flashforward, flashback or see it as of now... (24570 Bytes) » The Tom Kyte Blog
Oracle 9i exposed flashback query to developers for the first time.  The ability to flashback query dates back to version 4 however (it just wasn't exposed).  Every time you run a query in Oracle it is in fact a flashback query - it is what multi-versioning is all about.

However, there was never a flashforward query (well, ok, the workspace manager has this capability - but with lots of extra baggage).  We've never been able to ask a table "what will you look like tomorrow" - but now we do.

The capability is called Temporal Validity.  If you have a table with data that is effective dated - has a "start date" and "end date" column in it - we can now query it using flashback query like syntax.  The twist is - the date we "flashback" to can be in the future.  It works by rewriting the query to transparently the necessary where clause and filter out the right rows for the right period of time - and since you can have records whose start date is in the future - you can query a table and see what it would look like at some future time.

Here is a quick example, we'll start with a table:

ops$tkyte%ORA12CR1> create table addresses
  2  ( empno       number,
  3    addr_data   varchar2(30),
  4    start_date  date,
  5    end_date    date,
  6    period for valid(start_date,end_date)
  7  )
  8  /

Table created.



the new bit is on line 6 (it can be altered into an existing table - so any table  you have with a start/end date column will be a candidate).  The keyword is PERIOD, valid is an identifier I chose - it could have been foobar, valid just sounds nice in the query later.  You identify the columns in your table - or we can create them for you if they don't exist.  Then you just create some data:

ops$tkyte%ORA12CR1> insert into addresses (empno, addr_data, start_date, end_date )
  2  values ( 1234, '123 Main Street', trunc(sysdate-5), trunc(sysdate-2) );

1 row created.

ops$tkyte%ORA12CR1>
ops$tkyte%ORA12CR1> insert into addresses (empno, addr_data, start_date, end_date )
  2  values ( 1234, '456 Fleet Street', trunc(sysdate-1), trunc(sysdate+1) );

1 row created.

ops$tkyte%ORA12CR1>
ops$tkyte%ORA12CR1> insert into addresses (empno, addr_data, start_date, end_date )
  2  values ( 1234, '789 1st Ave', trunc(sysdate+2), null );

1 row created.


and you can either see all of the data:

ops$tkyte%ORA12CR1> select * from addresses;

     EMPNO ADDR_DATA                      START_DAT END_DATE
---------- ------------------------------ --------- ---------
      1234 123 Main Street                27-JUN-13 30-JUN-13
      1234 456 Fleet Street               01-JUL-13 03-JUL-13
      1234 789 1st Ave                    04-JUL-13

or query "as of" some point in time - as  you can see in the predicate section - it is just doing a query rewrite to automate the "where" filters:

ops$tkyte%ORA12CR1> select * from addresses as of period for valid sysdate-3;

     EMPNO ADDR_DATA                      START_DAT END_DATE
---------- ------------------------------ --------- ---------
      1234 123 Main Street                27-JUN-13 30-JUN-13

ops$tkyte%ORA12CR1> @plan
ops$tkyte%ORA12CR1> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------------
SQL_ID  cthtvvm0dxvva, child number 0
-------------------------------------
select * from addresses as of period for valid sysdate-3

Plan hash value: 3184888728

-------------------------------------------------------------------------------
| Id  | Operation         | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |           |       |       |     3 (100)|          |
|*  1 |  TABLE ACCESS FULL| ADDRESSES |     1 |    48 |     3   (0)| 00:00:01 |
-------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter((("T"."START_DATE" IS NULL OR
              "T"."START_DATE"<=SYSDATE@!-3) AND ("T"."END_DATE" IS NULL OR
              "T"."END_DATE">SYSDATE@!-3)))

Note
-----
   - dynamic statistics used: dynamic sampling (level=2)


24 rows selected.

ops$tkyte%ORA12CR1> select * from addresses as of period for valid sysdate;

     EMPNO ADDR_DATA                      START_DAT END_DATE
---------- ------------------------------ --------- ---------
      1234 456 Fleet Street               01-JUL-13 03-JUL-13

ops$tkyte%ORA12CR1> @plan
ops$tkyte%ORA12CR1> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------------
SQL_ID  26ubyhw9hgk7z, child number 0
-------------------------------------
select * from addresses as of period for valid sysdate

Plan hash value: 3184888728

-------------------------------------------------------------------------------
| Id  | Operation         | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |           |       |       |     3 (100)|          |
|*  1 |  TABLE ACCESS FULL| ADDRESSES |     1 |    48 |     3   (0)| 00:00:01 |
-------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter((("T"."START_DATE" IS NULL OR
              "T"."START_DATE"<=SYSDATE@!) AND ("T"."END_DATE" IS NULL OR
              "T"."END_DATE">SYSDATE@!)))

Note
-----
   - dynamic statistics used: dynamic sampling (level=2)


24 rows selected.

ops$tkyte%ORA12CR1> select * from addresses as of period for valid sysdate+3;

     EMPNO ADDR_DATA                      START_DAT END_DATE
---------- ------------------------------ --------- ---------
      1234 789 1st Ave                    04-JUL-13

ops$tkyte%ORA12CR1> @plan
ops$tkyte%ORA12CR1> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------------
SQL_ID  36bq7shnhc888, child number 0
-------------------------------------
select * from addresses as of period for valid sysdate+3

Plan hash value: 3184888728

-------------------------------------------------------------------------------
| Id  | Operation         | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |           |       |       |     3 (100)|          |
|*  1 |  TABLE ACCESS FULL| ADDRESSES |     1 |    48 |     3   (0)| 00:00:01 |
-------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter((("T"."START_DATE" IS NULL OR
              "T"."START_DATE"<=SYSDATE@!+3) AND ("T"."END_DATE" IS NULL OR
              "T"."END_DATE">SYSDATE@!+3)))

Note
-----
   - dynamic statistics used: dynamic sampling (level=2)


24 rows selected.


All in all a nice, easy way to query effective dated information as of a point in time without a complex where clause.  You need to maintain the data - it isn't that a delete will turn into an update the end dates a record or anything - but if you have tables with start/end dates, this will make it much easier to query them.

*Note added 4-jul-2013: this feature currently is not supported/working with the pluggable database infrastructure.  This is a temporary limitation.
06:30 12c - Whitelists... (6631 Bytes) » The Tom Kyte Blog
Starting in Oracle Database 12c - you can limit down to the package/procedure or function level what bits of code may invoke other bits of code in the database.  This process is called "white listing" and can be used to implement the concept of least privileges in your database.

In the past - if a given schema A had packages P1, P2, P3, ... Pn - then any of P1 .. Pn could invoke any function or procedure exposed in the specification of any of P1 .. Pn.  There would be no way to stop one bit of code from invoking any other bit of code.  This could have implications in the area of SQL Injection.  If one of the packages was subject to a SQL Injection bug - then that package could be used to execute ANY of the existing bits of code in that schema.  Additionally - even if none of the packages in that schema had a SQL Injection bug - but the application connected to the database itself did, an attacker could use that bug to execute any bit of code in that schema.

With the white list approach, the only way to execute a given piece of code would be to run it from a specific set of compiled units.  You cannot execute a white listed unit from the top level, it must be called by some specific set of units.  Now a SQL injection bug in the application cannot execute this code (it would have to call it as a top level call - but a white listed unit cannot be called that way).  And even further - a SQL injection bug in the code stored in the database will not be able to execute this white listed code (unless of course it was on the white list).

This is all accomplished with the new "accessible by" clause.  The use of this clause on a unit will restrict the calling set of units to be those in the accessible by clause and the unit itself (a units code is always accessible to itself).

For example, I'll create a package that is to be used only by procedure P1 (and itself) in some schema:

ops$tkyte%ORA12CR1> create or replace package my_pkg
2 accessible by (p1)
3 as
4 procedure p;
5 function f return number;
6 end;
7 /
Package created.
ops$tkyte%ORA12CR1> create or replace package body my_pkg
2 as
3
4 procedure p
5 is
6 begin
7 dbms_output.put_line( 'hello world' );
8 end;
9
10 function f return number
11 is
12 begin
13 p;
14 return 42;
15 end;
16
17 end;
18 /
Package body created.

as you can see - I've used the accessible by clause in the package specification.  This will restrict this package to be invoked only by procedure P1 or the package MY_PKG in this same schema. We can see that MY_PKG can invoke itself since function F calls procedure P in that same package.  Additionally - we can see that only procedure P1 outside of MY_PKG can invoke the functionality of this package. For example:

ops$tkyte%ORA12CR1> create or replace procedure p1
2 as
3 begin
4 my_pkg.p;
5 end;
6 /

Procedure created.

ops$tkyte%ORA12CR1> create or replace procedure p2
2 as
3 begin
4 my_pkg.p;
5 end;
6 /

Warning: Procedure created with compilation errors.

ops$tkyte%ORA12CR1> show errors
Errors for PROCEDURE P2:

LINE/COL ERROR
-------- -----------------------------------------------------------------
4/5 PL/SQL: Statement ignored
4/5 PLS-00904: insufficient privilege to access object MY_PKG


Procedure P1 successfully compiles and would be able to invoke MY_PKG.P but P2 cannot.  Furthermore, an attempt to execute MY_PKG as a top level call will fail:

ops$tkyte%ORA12CR1> exec my_pkg.p
BEGIN my_pkg.p; END;

*
ERROR at line 1:
ORA-06550: line 1, column 7:
PLS-00904: insufficient privilege to access object MY_PKG
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored


It should be noted that the acccessible by clause list is not evaluated at compile time for the unit being protected.  In the above example - we stated MY_PKG would be accessible by P1, before P1 was created.  That means you can put just about anything you want in the accessible by clause without raising an error (so be careful).

The accessible clause can be used across schemas as well.  If we recreate the package specification as:

ops$tkyte%ORA12CR1> create or replace package my_pkg
2 accessible by (p1,scott.p)
3 as
4 procedure p;
5 function f return number;
6 end;
7 /

Package created.
ops$tkyte%ORA12CR1> grant execute on my_pkg to scott;
Grant succeeded.

we'll be able to successfully compile and execute the code from SCOTT.P:

ops$tkyte%ORA12CR1> connect scott/tiger
Connected.
scott%ORA12CR1> create or replace procedure p
2 as
3 begin
4 ops$tkyte.my_pkg.p;
5 end;
6 /
Procedure created.

scott%ORA12CR1> exec p
hello world

PL/SQL procedure successfully completed.

scott%ORA12CR1> exec ops$tkyte.my_pkg.p
BEGIN ops$tkyte.my_pkg.p; END;

*
ERROR at line 1:
ORA-06550: line 1, column 7:
PLS-00904: insufficient privilege to access object MY_PKG
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored


but note that once again - SCOTT cannot invoke this package from the top level either - meaning if the SCOTT schema has some SQL injection issues - we've removed the ability for an attacker to invoke OPS$TKYTE.MY_PKG from that schema.

Next time I'll be taking a look at another new PL/SQL security feature - code based access control, the ability to grant ROLES to code...



06:30 12c - Utl_Call_Stack... (26384 Bytes) » The Tom Kyte Blog
Over the next couple of months, I'll be writing about some cool new little features of Oracle Database 12c - things that might not make the front page of Oracle.com.  I'm going to start with a new package - UTL_CALL_STACK.

In the past, developers have had access to three functions to try to figure out "where the heck am I in my code", they were:




Now these routines, while useful, were of somewhat limited use.  Let's look at the format_call_stack routine for a reason why.  Here is a procedure that will just print out the current call stack for us:

ops$tkyte%ORA12CR1> create or replace
  2  procedure Print_Call_Stack
  3  is
  4  begin
  5    DBMS_Output.Put_Line(DBMS_Utility.Format_Call_Stack());
  6  end;
  7  /
Procedure created.


Now, if we have a package - with nested functions and even duplicated function names:

ops$tkyte%ORA12CR1> create or replace
  2  package body Pkg is
  3    procedure p
  4    is
  5      procedure q
  6      is
  7        procedure r
  8        is
  9          procedure p is
 10          begin
 11            Print_Call_Stack();
 12            raise program_error;
 13          end p;
 14        begin
 15          p();
 16        end r;
 17      begin
 18        r();
 19      end q;
 20    begin
 21      q();
 22    end p;
 23  end Pkg;
 24  /

Package body created.

When we execute the procedure PKG.P - we'll see as a result:

ops$tkyte%ORA12CR1> exec pkg.p
----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0x6e891528         4  procedure OPS$TKYTE.PRINT_CALL_STACK
0x6ec4a7c0        10  package body OPS$TKYTE.PKG
0x6ec4a7c0        14  package body OPS$TKYTE.PKG
0x6ec4a7c0        17  package body OPS$TKYTE.PKG
0x6ec4a7c0        20  package body OPS$TKYTE.PKG
0x76439070         1  anonymous block

BEGIN pkg.p; END;

*
ERROR at line 1:
ORA-06501: PL/SQL: program error
ORA-06512: at "OPS$TKYTE.PKG", line 11
ORA-06512: at "OPS$TKYTE.PKG", line 14
ORA-06512: at "OPS$TKYTE.PKG", line 17
ORA-06512: at "OPS$TKYTE.PKG", line 20
ORA-06512: at line 1

The bit in red above is the output from format_call_stack whereas the bit in black is the error message returned to the client application (it would also be available to you via the format_error_backtrace API call). As you can see - it contains useful information but to use it you would need to parse it - and that can be trickier than it seems.  The format of those strings is not set in stone, they have changed over the years (I wrote the "who_am_i", "who_called_me" functions, I did that by parsing these strings - trust me, they change over time!).

Starting in 12c - we'll have structured access to the call stack and a series of API calls to interrogate this structure.  I'm going to rewrite the print_call_stack function as follows:

ops$tkyte%ORA12CR1> create or replace
 2  procedure Print_Call_Stack
  3  as
  4    Depth pls_integer := UTL_Call_Stack.Dynamic_Depth();
  5  
  6    procedure headers
  7    is
  8    begin
  9        dbms_output.put_line( 'Lexical   Depth   Line    Name' );
 10        dbms_output.put_line( 'Depth             Number      ' );
 11        dbms_output.put_line( '-------   -----   ----    ----' );
 12    end headers;
 13    procedure print
 14    is
 15    begin
 16        headers;
 17        for j in reverse 1..Depth loop
 18          DBMS_Output.Put_Line(
 19            rpad( utl_call_stack.lexical_depth(j), 10 ) ||
 20                    rpad( j, 7) ||
 21            rpad( To_Char(UTL_Call_Stack.Unit_Line(j), '99'), 9 ) ||
 22            UTL_Call_Stack.Concatenate_Subprogram
 23                       (UTL_Call_Stack.Subprogram(j)));
 24        end loop;
 25    end;
 26  begin
 27    print;
 28  end;
 29  /

Here we are able to figure out what 'depth' we are in the code (utl_call_stack.dynamic_depth) and then walk up the stack using a loop.  We will print out the lexical_depth, along with the line number within the unit we were executing plus - the unit name.  And not just any unit name, but the fully qualified, all of the way down to the subprogram name within a package.  Not only that - but down to the subprogram name within a subprogram name within a subprogram name.  For example - running the PKG.P procedure again results in:

ops$tkyte%ORA12CR1> exec pkg.p
Lexical   Depth   Line    Name
Depth             Number
-------   -----   ----    ----
1         6       20      PKG.P
2         5       17      PKG.P.Q
3         4       14      PKG.P.Q.R
4         3       10      PKG.P.Q.R.P
0         2       26      PRINT_CALL_STACK
1         1       17      PRINT_CALL_STACK.PRINT
BEGIN pkg.p; END;

*
ERROR at line 1:
ORA-06501: PL/SQL: program error
ORA-06512: at "OPS$TKYTE.PKG", line 11
ORA-06512: at "OPS$TKYTE.PKG", line 14
ORA-06512: at "OPS$TKYTE.PKG", line 17
ORA-06512: at "OPS$TKYTE.PKG", line 20
ORA-06512: at line 1


This time - we get much more than just a line number and a package name as we did previously with format_call_stack.  We not only got the line number and package (unit) name - we got the names of the subprograms - we can see that P called Q called R called P as nested subprograms.  Also note that we can see a 'truer' calling level with the lexical depth, we can see we "stepped" out of the package to call print_call_stack and that in turn called another nested subprogram.

This new package will be a nice addition to everyone's error logging packages.  Of course there are other functions in there to get owner names, the edition in effect when the code was executed and more. See UTL_CALL_STACK for all of the details.















06:30 12c - Silly little trick with invisibility... (8363 Bytes) » The Tom Kyte Blog
This is interesting, if you hide and then unhide a column - it will end up at the "end" of the table.  Consider:

ops$tkyte%ORA12CR1> create table t ( a int, b int, c int );
Table created.

ops$tkyte%ORA12CR1>
ops$tkyte%ORA12CR1> desc t;
 Name                                                  Null?    Type
 ----------------------------------------------------- -------- ------------------------------------
 A                                                              NUMBER(38)
 B                                                              NUMBER(38)
 C                                                              NUMBER(38)

ops$tkyte%ORA12CR1> alter table t modify (a invisible);
Table altered.

ops$tkyte%ORA12CR1> alter table t modify (a visible);
Table altered.

ops$tkyte%ORA12CR1> desc t;
 Name                                                  Null?    Type
 ----------------------------------------------------- -------- ------------------------------------
 B                                                              NUMBER(38)
 C                                                              NUMBER(38)
 A                                                              NUMBER(38)


Now, that means you can add a column or shuffle them around.  What if we had just added A to the table and really really wanted A to be first.  My first approach would be "that is what editioning views are great at".  If I couldn't use an editioning view for whatever reason - we could shuffle the columns:

ops$tkyte%ORA12CR1> alter table t modify (b invisible);
Table altered.

ops$tkyte%ORA12CR1> alter table t modify (c invisible);
Table altered.

ops$tkyte%ORA12CR1> alter table t modify (b visible);
Table altered.

ops$tkyte%ORA12CR1> alter table t modify (c visible);
Table altered.

ops$tkyte%ORA12CR1>
ops$tkyte%ORA12CR1> desc t;
 Name                                                  Null?    Type
 ----------------------------------------------------- -------- ------------------------------------
 A                                                              NUMBER(38)
 B                                                              NUMBER(38)
 C                                                              NUMBER(38)


Note: that could cause some serious invalidations in your database - so make sure you are a) aware of that b) willing to pay that penalty and c) really really really want A to be first in the table!
06:30 12c - SQL Text Expansion (10385 Bytes) » The Tom Kyte Blog
Here is another small but very useful new feature in Oracle Database 12c - SQL Text Expansion.  It will come in handy in two cases:

  1. You are asked to tune what looks like a simple query - maybe a two table join with simple predicates.  But it turns out the two tables are each views of views of views and so on... In other words, you've been asked to 'tune' a 15 page query, not a two liner.
  2. You are asked to take a look at a query against tables with VPD (virtual private database) policies.  In order words, you have no idea what you are trying to 'tune'.
A new function, EXPAND_SQL_TEXT, in the DBMS_UTILITY package makes seeing what the "real" SQL is quite easy. For example - take the common view ALL_USERS - we can now:

ops$tkyte%ORA12CR1> variable x clob

ops$tkyte%ORA12CR1> begin
  2          dbms_utility.expand_sql_text
  3          ( input_sql_text => 'select * from all_users',
  4            output_sql_text => :x );
  5  end;
  6  /

PL/SQL procedure successfully completed.

ops$tkyte%ORA12CR1> print x

X
--------------------------------------------------------------------------------
SELECT "A1"."USERNAME" "USERNAME","A1"."USER_ID" "USER_ID","A1"."CREATED" "CREAT
ED","A1"."COMMON" "COMMON" FROM  (SELECT "A4"."NAME" "USERNAME","A4"."USER#" "US
ER_ID","A4"."CTIME" "CREATED",DECODE(BITAND("A4"."SPARE1",128),128,'YES','NO') "
COMMON" FROM "SYS"."USER$" "A4","SYS"."TS$" "A3","SYS"."TS$" "A2" WHERE "A4"."DA
TATS#"="A3"."TS#" AND "A4"."TEMPTS#"="A2"."TS#" AND "A4"."TYPE#"=1) "A1"

Now it is easy to see what query is really being executed at runtime - regardless of how many views of views you might have.  You can see the expanded text - and that will probably lead you to the conclusion that maybe that 27 table join to 25 tables you don't even care about might better be written as a two table join.

Further, if you've ever tried to figure out what a VPD policy might be doing to your SQL, you know it was hard to do at best.  Christian Antognini wrote up a way to sort of see it - but you never get to see the entire SQL statement: http://www.antognini.ch/2010/02/tracing-vpd-predicates/.  But now with this function - it becomes rather trivial to see the expanded SQL - after the VPD has been applied.  We can see this by setting up a small table with a VPD policy 

ops$tkyte%ORA12CR1> create table my_table
  2  (  data        varchar2(30),
  3     OWNER       varchar2(30) default USER
  4  )
  5  /
Table created.

ops$tkyte%ORA12CR1> create or replace
  2  function my_security_function( p_schema in varchar2,
  3                                 p_object in varchar2 )
  4  return varchar2
  5  as
  6  begin
  7     return 'owner = USER';
  8  end;
  9  /
Function created.

ops$tkyte%ORA12CR1> begin
  2     dbms_rls.add_policy
  3     ( object_schema   => user,
  4       object_name     => 'MY_TABLE',
  5       policy_name     => 'MY_POLICY',
  6       function_schema => user,
  7       policy_function => 'My_Security_Function',
  8       statement_types => 'select, insert, update, delete' ,
  9       update_check    => TRUE );
 10  end;
 11  /
PL/SQL procedure successfully completed.

And then expanding a query against it:

ops$tkyte%ORA12CR1> begin
  2          dbms_utility.expand_sql_text
  3          ( input_sql_text => 'select * from my_table',
  4            output_sql_text => :x );
  5  end;
  6  /
PL/SQL procedure successfully completed.

ops$tkyte%ORA12CR1> print x

X
--------------------------------------------------------------------------------
SELECT "A1"."DATA" "DATA","A1"."OWNER" "OWNER" FROM  (SELECT "A2"."DATA" "DATA",
"A2"."OWNER" "OWNER" FROM "OPS$TKYTE"."MY_TABLE" "A2" WHERE "A2"."OWNER"=USER@!)
 "A1"

Not an earth shattering new feature - but extremely useful in certain cases.  I know I'll be using it when someone asks me to look at a query that looks simple but has a twenty page plan associated with it!
06:30 12c - SQL Plus new things.... (3863 Bytes) » The Tom Kyte Blog
SQL Plus is still my typical tool of choice to "talk" to Oracle - 25+ years and going...  Oracle Database 12c has introduced a few new things in this venerable old tool

Last Login Time

$ sqlplus /

SQL*Plus: Release 12.1.0.1.0 Production on Sun Jul 7 13:53:15 2013

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Last Successful login time: Wed Jul 03 2013 14:30:14 -04:00

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

ops$tkyte%ORA12CR1>

By default, SQL Plus will display your last login time - if you don't want that, just use -nologintime.

More on invisible columns

I recently wrote about invisible columns and how they wouldn't be displayed via a DESCRIBE command.  That is - SQL Plus won't show them by default.  However, there is a SET command that will display them:

ops$tkyte%ORA12CR1> create table t
2 ( x int,
3 y int invisible
4 );

Table created.

ops$tkyte%ORA12CR1> desc t
Name Null? Type
---------------------------------------- -------- ----------------------------
X NUMBER(38)

ops$tkyte%ORA12CR1> show colinvisible
colinvisible OFF
ops$tkyte%ORA12CR1> set colinvisible ON
ops$tkyte%ORA12CR1> desc t
Name Null? Type
---------------------------------------- -------- ----------------------------
X NUMBER(38)
Y (INVISIBLE) NUMBER(38)


Support for Pluggable Databases

There is support for starting pluggable databases from a container database, as well as three new SHOW commands to see what pluggable databases there are and information about the current pluggable database you are connected to:

sys%ORCL> startup pluggable database pdborcl;
Pluggable Database opened.

sys%ORCL> show pdbs

CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 PDBORCL READ WRITE NO



sys%ORCL> connect scott/tiger@pdborcl
Connected.

scott%PDBORCL> show con_id

CON_ID
------------------------------
3

scott%PDBORCL> show con_name

CON_NAME
------------------------------
PDBORCL

For more details...


For more details - check out the SQL Plus guide.  It is always good to look through the existing commands anyway - to remind you of something you forgot you already knew (I do that all a lot - forget things I knew once upon  a time.  Just looking at the table of contents for SQL Plus could remind you of many forgotten SHOW and SET commands!)

06:30 12c - Multiple same column indexes... (4850 Bytes) » The Tom Kyte Blog
Another new capability of Oracle Database 12c is the ability to create more than one index on the same set of attributes.  That is - you can have two or more indexes on the column "X", or the set of columns "A,B,C".

So, that would beg the question "why - why would you want to do that?".  The answer is twofold - testing and availability.  The testing part is obvious - you might want to test how the performance of a bitmap index would be in your reporting system as compared to a b*tree index on the same set of columns.  You can now create that bitmap index even though the b*tree exists, test it out in your session, and then decide to get rid of the b*tree index in favor of the bitmap (or get rid of the bitmap of course).

The availability part isn't as obvious perhaps.  If you wanted to roll out a change to production that would replace a normal b*tree index with a reverse key index - that would have called for an outage in the past.  You would have to drop the original index and then create the new one.  During the period of time the index was initially dropped and the new one completed work, it is likely the applications that depended on that index for data retrieval would have to be offlined.  If they were not - their performance, in light of the missing index, could be disastrous - in effect, a denial of service attack on the database.

In Oracle Database 11g and before - this is what we would expect if we attempted to create more than one index on the same set of attributes:

ops$tkyte%ORA11GR2> create table t ( x int, y int, z int );
Table created.

ops$tkyte%ORA11GR2> create index t_idx on t(x,y);
Index created.

ops$tkyte%ORA11GR2> create bitmap index t_idx2 on t(x,y);
create bitmap index t_idx2 on t(x,y)
*
ERROR at line 1:
ORA-01408: such column list already indexed

ops$tkyte%ORA11GR2> create bitmap index t_idx2 on t(x,y) invisible;
create bitmap index t_idx2 on t(x,y) invisible
*
ERROR at line 1:
ORA-01408: such column list already indexed


as you can see - it just wasn't going to happen.  But now in 12c - as long as only one index is "visible" - we can create multiple indexes:

ops$tkyte%ORA12CR1> create table t ( x int, y int, z int );
Table created.

ops$tkyte%ORA12CR1> create index t_idx on t(x,y);
Index created.

ops$tkyte%ORA12CR1> create bitmap index t_idx2 on t(x,y) invisible;
Index created.

Now that the second index is there, we can test it - to evaluate the performance for example - easily:

ops$tkyte%ORA12CR1> alter session set optimizer_use_invisible_indexes=true;

Session altered.

ops$tkyte%ORA12CR1> exec dbms_stats.set_table_stats( user, 'T', numrows => 1000000, numblks => 100000 );

PL/SQL procedure successfully completed.

ops$tkyte%ORA12CR1> set autotrace traceonly explain
ops$tkyte%ORA12CR1> select count(*) from t;

Execution Plan
----------------------------------------------------------
Plan hash value: 1106681275

---------------------------------------------------------------------
| Id | Operation | Name | Rows | Cost (%CPU)|
---------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 0 (0)|
| 1 | SORT AGGREGATE | | 1 | |
| 2 | BITMAP CONVERSION COUNT | | 1000K| |
| 3 | BITMAP INDEX FAST FULL SCAN| T_IDX2 | | |
---------------------------------------------------------------------

ops$tkyte%ORA12CR1> set autotrace off

if we deemed that index to be what we really wanted - we would just drop the old index and make this one visible, or set the old one invisible and set this one visible - publish it in effect.

See http://docs.oracle.com/cd/E16655_01/server.121/e17906/chapter1.htm#FEATURENO09740 and especially http://richardfoote.wordpress.com/2013/07/02/12c-intro-to-multiple-indexes-on-same-column-list-repetition/ for more info.  (and subscribe to Richard's blog if you aren't already - best index information out there)


06:30 12c - Invisible Columns... (9565 Bytes) » The Tom Kyte Blog
Remember when 11g first came out and we had "invisible indexes"?  It seemed like a confusing feature - indexes that would be maintained by modifications (hence slowing them down), but would not be used by queries (hence never speeding them up).  But - after you looked at them a while, you could see how they can be useful.  For example - to add an index in a running production system, an index used by the next version of the code to be introduced later that week - but not tested against the queries in version one of the application in place now.  We all know that when you add an index - one of three things can happen - a given query will go much faster, it won't affect a given query at all, or... It will make some untested query go much much slower than it used to.  So - invisible indexes allowed us to modify the schema in a 'safe' manner - hiding the change until we were ready for it.

Invisible columns accomplish the same thing - the ability to introduce a change while minimizing any negative side effects of that change.  Normally when you add a column to a table - any program with a SELECT * would start seeing that column, and programs with an INSERT INTO T VALUES (...) would pretty much immediately break (an INSERT without a list of columns in it).  Now we can add a column to a table in an invisible fashion, the column will not show up in a DESCRIBE command in SQL*Plus, it will not be returned with a SELECT *, it will not be considered in an INSERT INTO T VALUES statement.  It can be accessed by any query that asks for it, it can be populated by an INSERT statement that references it, but you won't see it otherwise.

For example, let's start with a simple two column table:

ops$tkyte%ORA12CR1> create table t
  2  ( x int,
  3    y int
  4  )
  5  /
Table created.

ops$tkyte%ORA12CR1> insert into t values ( 1, 2 );
1 row created.

Now, we will add an invisible column to it:

ops$tkyte%ORA12CR1> alter table t add 
                    ( z int INVISIBLE );
Table altered.

Notice that a DESCRIBE will not show us this column:

ops$tkyte%ORA12CR1> desc t
 Name              Null?    Type
 ----------------- -------- ------------
 X                          NUMBER(38)
 Y                          NUMBER(38)

and existing inserts are unaffected by it:

ops$tkyte%ORA12CR1> insert into t values ( 3, 4 );
1 row created.

A SELECT * won't see it either:

ops$tkyte%ORA12CR1> select * from t;

         X          Y
---------- ----------
         1          2
         3          4

But we have full access to it (in well written programs! The ones that use a column list in the insert and select - never relying on "defaults":

ops$tkyte%ORA12CR1> insert into t (x,y,z) 
                        values ( 5,6,7 );
1 row created.

ops$tkyte%ORA12CR1> select x, y, z from t;
         X          Y          Z
---------- ---------- ----------
         1          2
         3          4
         5          6          7

and when we are sure that we are ready to go with this column, we can just modify it:

ops$tkyte%ORA12CR1> alter table t modify z visible;
Table altered.

ops$tkyte%ORA12CR1> select * from t;

         X          Y          Z
---------- ---------- ----------
         1          2
         3          4
         5          6          7


I will say that a better approach to this - one that is available in 11gR2 and above - would be to use editioning views (part of Edition Based Redefinition - EBR ).  I would rather use EBR over this approach, but in an environment where EBR is not being used, or the editioning views are not in place, this will achieve much the same.

Read these for information on EBR:


06:30 12c - Implicit Result Sets... (9032 Bytes) » The Tom Kyte Blog
Since version 7.2 of Oracle we've been able to return result sets from stored procedures to clients.  The way we've accomplished this in the past is via a REF CURSOR.  I've always liked this approach since it makes it so that the signature of a stored procedure/function makes it clear that a procedure/function a) actually returns a result set and b) can be done such that the 'shape' of the result set is known at compile time.

What that means is - the REF CURSOR would be a formal named parameter for the procedure or the return value of a function.  If you describe the procedure - you'll see it.  You'll know exactly how many result sets the procedure returns and you have the capability to know the number of columns, names of columns and datatypes if the developer chose to use strongly typed ref cursors.  In other words - you'll know what you are getting, the REF CURSORS are explicitly there, staring you in the face.

Other databases adopted an implicit approach.  If you describe their procedures - you will have no idea how many or what kind of result sets they return.  You have to run them and see what they decide to send back - and each time you run them, they could return a different result.  In other words, you had to read the code to see what results might be coming back to you.  I'm not a huge fan of this approach - it is much less self documenting, more error prone (in my opinion).

That said - this difference (explicit versus implicit result sets) can make migrating an application from these databases to Oracle difficult.  You have to change the inputs/outputs of your stored procedures - adding the ref cursors and you have to modify the client code.  You cannot just change the stored procedure, you have to change the client.  When using implicit result sets - the client code would look similar to this psuedo code:

execute stored procedure( param1, param2, .. paramN )
while more result-sets loop
   while more-data-from-that-result-set loop
       process data
   end while more-data-from-that-result-set
end while result-sets

while the code for explict result sets would resemble:

execute stored procedure
( param1, param2, ... paramN, 
  result-set1, result-set2, ... result-setN)

while more-data-in-result-set1 loop
    process data
end while more-data-in-result-set1
...
while more-data-in-result-setN loop
    process data
end while more-data-in-result-setN

so, the structure of the client code must be modified in addition to the stored procedures signature (it's inputs and outputs).  

In order to ease migrations from implicit result set databases to Oracle, Oracle database 12c introduced support for implicit result sets - result sets that can be returned to a client but do not necessitate a formal named parameter.  The client code to process such a result set is identical now - the client code needs not change, the stored procedure signature need not change - the body of the procedure just needs to be implemented in PL/SQL.

We accomplish this magic in PL/SQL via two new API calls in the DBMS_SQL package - one for returning REF CURSORS and one for returning DBMS_SQL cursors.  They are:



PROCEDURE RETURN_RESULT(rc IN OUT SYS_REFCURSOR, to_client IN BOOLEAN DEFAULT TRUE); 
PROCEDURE RETURN_RESULT(rc IN OUT INTEGER, to_client IN BOOLEAN DEFAULT TRUE);

TO_CLIENT => true, return to client layer
TO_CLIENT=> false, return to invoker, immediate caller - could be another plsql routine


to demonstrate this, we can use SQL*Plus as the client and show an implicit result set being returned to the client and auto-magically printed (this will only work with a 12c SQL*Plus!! it is the first SQL*Plus to recognize that there might be result sets to print)


ops$tkyte%ORA12CR1> declare
2 c1 sys_refcursor;
3 c2 sys_refcursor;
4 begin
5 open c1 for select * from dept;
6 dbms_sql.return_result(c1);
7
8 open c2 for select * from dual;
9 dbms_sql.return_result(c2);
10 end;
11 /

PL/SQL procedure successfully completed.

ResultSet #1

DEPTNO DNAME LOC
---------- -------------- -------------
10 ACCOUNTING NEW YORK
20 RESEARCH DALLAS
30 SALES CHICAGO
40 OPERATIONS BOSTON

ResultSet #2

D
-
X

ops$tkyte%ORA12CR1>

as you can see - there were no inputs/outputs to this block of code - we implicitlly returned the two result sets using the new DBMS_SQL API and SQL*Plus used a method that would result the second set of psuedo code above to discover what results might be available and then print them out.

We can process these implicit result sets in PL/SQL if we want (they don't have to be returned to a client - the ref cursor/dbms_sql cursor can be implicitly returned to a PL/SQL function/procedure) using the two new API calls:

PROCEDURE GET_NEXT_RESULT(c IN INTEGER, rc OUT SYS_REFCURSOR);
PROCEDURE GET_NEXT_RESULT(c IN INTEGER, rc OUT INTEGER);

(see http://docs.oracle.com/cd/E16655_01/appdev.121/e17622/dynamic.htm#LNPLS2176 for a description and example of this API).

But in general, it'll be a client bit of code that processes these result sets and a client java program might resemble:

Connection conn = DriverManager.getConnection(jdbcURL, user, password);
try
{
Statement stmt = conn.createStatement ();
stmt.executeQuery ( “begin foo; end;” );

while (stmt.getMoreResults())
{
ResultSet rs = stmt.getResultSet();
System.out.println("ResultSet");
while (rs.next())
{
/* get results */
}
}
}

I see Tim Hall has written about this recently as well - you can see his writeup here: http://www.oracle-base.com/articles/12c/implicit-statement-results-12cr1.php



06:30 12c - Code Based Access Control (CBAC) part 1 (6714 Bytes) » The Tom Kyte Blog
One frequently asked question posed by stored procedure developers since version 7.0 was released in 1992 has been "why do I get an ora-1031 in PL/SQL but not in SQL*Plus directly".  I get this question on asktom a lot, I've written articles about it in Oracle magazine, I must have explained this thousands of times by now.

And now, it all has to change - roles and stored procedures are no longer like matter and anti-matter.  Staring in Oracle Database 12c a role may be granted to a bit of code, and that role will be active only while that procedure is executing.  Think about what that means - you can have a schema chock full of code - hundreds of packages - but have only one package that has a certain privilege at run time.  This will allow you to implement the concept of "least privileges" fully.  What could be more least privileges than granting a privilege to a specific bit of code?

Think about this from a SQL injection protection point of view.  You could either grant privilege X directly to the schema  - meaning this privilege would be available for every single stored unit in that schema to use at anytime, or you can grant this privilege to a role and then grant that role to that unit.  That unit and only that unit would be able to use privilege X at runtime.  If some other units in that schema had a SQL injection bug - they would not be able to utilize that privilege.

We'll take a look at this new capability from two perspectives - from that of a definers rights routine (the default) and from that of an invokers rights routine.  In the case of the definers rights routine, this new capability will only make sense when you use dynamic SQL.  In the case of the invokers rights routine, this new capability has a much larger impact and makes the use of invokers rights routines much wider than it was in the past.  I'll defer talking more about invokers rights routines until next time and we'll concentrate on definers rights routines for now.

Definers rights routines compile with the set of privileges granted directly to the owner of the procedure - roles are never enabled during the compilation of a compiled stored object.  This is true in Oracle Database 12c still - and is the reason this new capability only makes sense with dynamic SQL in a definers rights routine.  In order for the unit to compile, all of the privileges necessary for the static SQL and PL/SQL in the unit must be granted directly to the owner of the unit.  Therefore - any privileges granted via roles cannot be used for static SQL or PL/SQL.  The compilation would fail without the direct privilege.  However, any dynamically executed code would not be security checked until runtime, the compiler would not "see" this code.  And with CBAC - the set of privileges the dynamic SQL will be checked with will be all of the privileges granted directly to the owner of the unit and any privileges associated with roles granted to the unit.

So, if we start with a simple user and role:

ops$tkyte%ORA12CR1> create user a identified by a
2 default tablespace users
3 quota unlimited on users;
User created.
ops$tkyte%ORA12CR1> create role create_table_role;
Role created.

and then we grant some privileges to the user and the role:

ops$tkyte%ORA12CR1> grant create table to create_table_role;
Grant succeeded.

ops$tkyte%ORA12CR1> grant create session, create procedure to a;
Grant succeeded.

ops$tkyte%ORA12CR1> grant create_table_role to a with admin option;
Grant succeeded.

ops$tkyte%ORA12CR1> alter user a default role all except create_table_role;
User altered.

we are ready to start. Note that the user A has only the create session and create procedure privilege granted to the directly. They do have the CREATE TABLE privilege, but that privilege is granted via a role to the user - it will not be available to that user during the compilation of a stored unit, nor would it be available at runtime (until we grant it to the code itself).

So, let's create a procedure in this account:

ops$tkyte%ORA12CR1> connect a/a
Connected.
a%ORA12CR1> create or replace procedure p
2 as
3 begin
4 execute immediate
5 'create table t ( x int )';
6 end;
7 /
Procedure created.

Now, the procedure created successfully since we had the create procedure privilege, but if you try to run it you would receive:

a%ORA12CR1> exec p
BEGIN p; END;

*
ERROR at line 1:
ORA-01031: insufficient privileges
ORA-06512: at "A.P", line 4
ORA-06512: at line 1

and in fact in 11g and before - that would be the only thing you would ever receive.  You would have to grant CREATE TABLE to the schema A - making it available to every single stored unit in that schema.  But in Oracle Database 12c - we can grant the CREATE_TABLE_ROLE to the procedure:

a%ORA12CR1> set role create_table_role;
Role set.

a%ORA12CR1> grant create_table_role to procedure p;
Grant succeeded.

a%ORA12CR1>
a%ORA12CR1> exec p

PL/SQL procedure successfully completed.

a%ORA12CR1> set linesize 40
a%ORA12CR1> desc t
Name Null? Type
----------------- -------- ------------
X NUMBER(38)

the CREATE TABLE privilege is now available to the stored procedure P and only that stored procedure.  No other bits of code in this schema would be able to create a table.

So, in short, dynamic SQL and PL/SQL executed within a definers rights routine can now take advantage of privileges granted to roles.  This will allow you to implement the concept of "least privileges" (and to use roles in definers rights routines).

In the next article, we'll look at this from the perspective of an invokers rights routine.  That is where this new capability gets really interesting!

2018-10-17 Wed

23:11 Problem Solving » Oracle Scratchpad

2018-10-15 Mon

20:37 Faking Histograms » Oracle Scratchpad