I agree, but I think Herb's point is more than what can be solved by having smarter JITers. A smarter compiler can figure out parallel exection paths, but that applies only to a single thread of execution; I would hazard a guess that most contemporary compilers are already very good at this. When it comes to multiple threads however, there is not much a compiler or a JITer can do. JITers can't make your multithreaded code run faster beyond the optmization they do already, because the synchronization semantics are understood only at the application level. In this case, the programmer simply has to get better at concurrent programming -- something that is very hard indeed. Most programs need to do disk or network I/O quite often, so programmers will need to do more multi-threaded stuff at the application level if CPU speeds are not going to keep on increasing.

The other alternative (and I think something that shows a lot of promise, if enough investment is made) is to develop new languages with better and safer concurrent programming models. The academics have been working in this area for a while, but we need more of these things to come into the mainstream. One example is Erlang which is a language developed by a research lab within Ericsson. It eschews shared shared memory in favour of message-passing concurrency; while that sounds limiting, they have been to write huge concurrent systems safely by making the cost of process creation extremely small.

Regarding your last point about the meaning of CPU bound, I did not understand which ("the latter") you meant as being the bigger problem: CPU crunching or cache misses? Stalling for code or going to main memory due to a cache miss is going to be a problem, I agree -- however this is something they can improve even with processor speeds capping out by continuing to increase cache sizes, at least to a certain extent.

In the last few years, the rate at which network speeds have been increasing has exceeded the rate at which CPU speeds have been increasing. A statistic that I read recently said that in the last decade processor speeds have been increasing by a factor of 100 every decade, while disk capacities have been increasing by a factor or 1000 and optical network bandwidth by a factor of 10000. These are exponentials at work here -- it won't be long before network access will be almost as fast as main memory access. In fact, I am pretty sure that network access will be faster than accessing your local disk. This reversal of trends will have a drastic effect on common assumptions and trade-offs that programmers have traditionally made. Such a shift in mindset is difficult to come to grips with. For instance, it is true today that on high-end processors floating-point multiplications are faster than integer multiply operations, yet many oldtime programmer (including me) instinctively try to do most arithmetic with integers -- it is just hard to change the mindset. If/when the "CPU cycles are cheap, network access is costly" assumption goes out the window, a lot of things will have to change. For instance, as I mentioned to Fario the other day, codec research may end up being a dead-end.