Rob Knop discusses his woes trying to get tenure at Vanderbilt. By all accounts, Rob is good at his job (teaches well, serves on committees, etc) except for getting grants. That will be his downfall. His point, if NSF is only funding 16-20% of all the grants submitted there are too many people trying to be faculty members (this is in astronomy). It is overly competitive, selecting people like him out. Some like Chad have pointed out that all the depressing news of tenure can be a negative, encouraging students not to pursue graduate studies. The key is going in with your eyes wide-open. The advice Chad got as an undergraduate was "the only reason to go to graduate school was in order to pursue a career in research-- which, it should be noted, is not identically equal to academia." (The latter is why it is important to choose an advisor in graduate school who is open to you exploring options besides being a faculty member at a research university).
Incoherently Scattered Ponderings and Open Reading Frame point to the fact that the bottleneck is not getting tenure per se but rather getting a tenure track position to begin with, the jump from being a post-doc. It is also why so many advisors can be such jerks. They figure if you leave, there is someone else who will be willing to jump through their hoops. Of course this selects for people in the sciences who are overly passionate about science relative to other facets of life. And when those people choose who goes to graduate school, they pick people like themselves-focussed on science and the cycle keeps going.
Of course what feeds this push for more post-docs/grad students than there will positions for? For faculty members to focus on getting grants and why it is so important for getting tenure? The dollars brought in from grants. Mike the Mad Biologist points this out in his skepticism of calls in Harvard for improved teaching leading to anything. Why? The selection is for money which comes from doing research and not from teaching.
"Overhead, also referred to as indirect costs, are a surcharge on the direct or actual costs* of the grant. More people on a grant and more research costs mean more 'indirects' for the institution. Typically, these indirects run 50-75% of direct costs."
"A certain amount of indirects is needed: all institutions have administrative and infrastructure costs (e.g., personnel, IT, utilities, and so on). But 50%-75% is exorbitant (and, incidentally, reduces the total number of awards federal agencies can give. Federal granting agencies subsidize higher education to the tune of billions of dollars every year***)."
In other words, we are funding higher education in the United States through science research grants. Universities get even more money from the government by charging PhD students tuition which is paid for by those federal grants. The tuition for PhD students in the humanities and social studies is usually waived (it was at GradU and it is waived at SnobU). SnobU by the way charges for internet access and email accounts in addition to the overhead so it takes in even more revenue. Universities are addicted to this grant money. It helps pay for administrative costs. It pays for non-revenue producing departments. It allows universities to grow their endowments which in turn makes them more prestigious attracting more students, allowing the university to be more selective and thus more prestigious. Faculty thus are under pressure therefore to get grants and keep getting them. The system puts a strong selection on keeping labor costs low (hence very few faculty positions relative to the number post-docs/graduate students) since so much grant money goes towards overhead but a lot of research is needed in order to get a grant. This sets up a situation that favors bringing more graduate students and post-docs into the system than their will be jobs for as faculty members. It also favors advisors trying to get every last drop out of those in their labs which is why so many push their post-docs and graduate students to work long hours.
The overabundance of PhD candidates & newly minted PhDs in turn creates an overly competitive environment between post-docs and graduate students, who for the most part are encouraged to shoot for being professors at research universities. It is why so many young scientists are so willing to put up with such horrible working conditions. The chances of getting a tenure track position are low. Getting a slightly better letter of rec. from your advisor becomes vital for your future.
The question is how do you break this cycle that feeds itself? Reducing overhead at this stage won't solve the problem because universities will just put greater pressure on advisors to get grants. More money will help but it can't just be in the shape of research grants. There has to be incentives for teaching well. For being good mentors. How do you put that into grants? It will also require graduate students and post-docs being more involved in how their departments are run, having a real voice and say. That is scary not just to administrators but also faculty.
Of course that would require these young scientists to organize. This seems unlikely as the competitive environment selects out those who would be most inclined towards this (Incoherent has a post on this centered around the number of scientists who played team sports as kids). Not to mention the current organizing seems to be centered around forming traditional employee unions. For graduate students especially, I think this would be a mistake. Being treated like employees is the problem. The fight has to be for fighting to maintain their status as students who demand quality teaching and mentorship. Who demand conditions that allow them to learn (which at this age group includes such things as child care). That of course requires revolution and effort but what choice do we really have? Are we really getting the best science for our dollars? What type of scientists are being churned out? Is this why scientists have trouble connecting to the general populace?