Let's go back in time to the 19th century. Railways were initially built by private companies and run as independent businesses. Even then, though, there was some level of state involvement, as land acquisition was hard, and the railways relied on private Acts of Parliament to compulsorily purchase the land. Without these Private Acts there could have been no long-distance railways. These companies were run as for-profit concerns, but there was a "railway bubble", and lots of money was wasted building routes that could never be economic, or duplicated other routes. People lost their savings.
Step forward half a century, and you have the railway industry just before the first world war, which you'd think of as the peak. However, many railway companies were losing money. After the war the railways were Grouped - which is to say that Parliament amalgamated them into four large companies (the Great Western, the London, Midland and Scottish, the London and North Eastern, and Southern).
By the 1940s, it was clear that that motor vehicles were leading to a paradigm shift in transport. In 1948, the four companies were suffering badly (arguably due to the war), and were nationalised as British Railways, by Clement Atlee's socialist Labour government. The railways continued to lose money into the 1950s. The Modernisation Plan of 1955 made some important changes (such as the withdrawal of steam), but made several miscalls, in particular seeing a future in wagonload freight.
It is in this context that we come to Mr. Beeching, who was tasked by Harold Macmillan's government with cutting the losses in an industry which was by now assumed to be on a permanent decline. He found the lines that were most unprofitable when considered on their own, and closed them. In doing so, the network effect was broken. No more was the railway a way of getting from point-to-point anywhere in the country, and faced with the choice of driving to their nearest railhead and taking the train, or just driving to their destination, many people just chose the latter - especially given the investment in motorways. But eventually, a strange thing happened - this decline levelled out. This was a combination of roads now becoming congested, and initiatives like the InterCity 125, which made long journeys massively faster by rail.
British Rail continued ascendant in the 1980s, with an internal reorganisation ("Sectorisation"), electrification of lines, the introduction of new standardised trains for suburban operation, and an increase in passenger numbers while its subsidy remained level. Government even saw the benefit of urban railways in regeneration, with improvements to Merseyrail, the North London Line, and the creation of the DLR.
And then John Major came. Now, privatisation was fairly fashionable in the 1980s, but Thatcher left the railways well alone. I'm old enough to remember the bitter fighting over the privatisation, and I remember a particular concern being the structure the industry would have.
To replace British Railways - which is ostensibly being privatised because it is a limbering bureaucratic monstrosity - it was instead split into 1 infrastructure owner, 3 rolling stock companies (ROSCOs), and 25 train operating companies (TOCs), along with various other players, all of whom had complex contractual arrangements with each other, which had to be synthesised out of thin air rather than evolving out of best practice or informed negotiation. The reason for there being so many train companies is "competition". Now, it's easy to mock this and many have, as most journeys are monopolies - but they are not supposed to be competing with each other for passengers. They are supposed to be competing against each other for franchises. Every few years, they bid for control over a set of routes, and the DfT picks the option that conforms to a minimum standard of service while producing the best return for the exchequer. This is all very well unless the winning bid was so overoptimistic in their predictions they can't deliver and end up giving the keys back early. This has happened twice on the ECML now, which led to the formation of East Coast (run by the ironically named arms-length Directly Operated Railways).
It was promised that private money would be able to provide new investment, but in fact the ongoing replacement of old stock with shiny new trains was halted for two years. The infrastructure owner, Railtrack, was floated in spring 1996, which raised £1.9 billion pounds. The stock price rose by 300% over the next two years (and though there was an economy recovery and rise in the FTSE 100 during this period, it was nothing comparable), apparently because it starved tracks of needed maintenance money in order to issue large dividends, safe in the knowledge that the government would bail it out.
We know how that one ended: it didn't. But while the infrastructure has been brought under effective state control as Network Rail, the costs still suffer from years of "railway inflation" (or, as Modern Railways puts it, "boiling frog syndrome"), at well in excess of RPI. The government explained this in 2006 as the result of contractors being able to negotiate better prices. Each big project is divided into different contracts, each bit competing against each other to get the contractors. A proper monopsony would have better buying power, and it is for that reason, and others that Network Rail took maintenance in-house. But even after years of paying over the odds, it's difficult politically to force skilled workers within a sector to take a pay cut (especially if they are working overtime and odd hours in order that you can get the work done out of working hours, so that you don't have to pay compensation to other bits of the industry.)
But, we have, for all its faults, a dynamic new railway network today, don't we? So privatisation must have succeeded, in the end? We see modern trains appearing on our railway lines, new railway stations open, sometimes even old lines reopening, smartcard ticketing, the transformation of the old Silverlink suburban network in London. And passenger numbers are booming!
Except that the state has been the source of every single one of these innovations - TOCs only run new services when they are included in their bid, the DfT has to underwrite the trains because the rolling stock companies are afraid to, local governments have paid for the station reopenings and new lines, and Oyster and the London Overground were driven by TfL and Ken Livingstone. Passengers are only using the railways in greater numbers because the alternatives are also awful. Schemes to improve public transport are sometimes avoided because they would be too popular - witness the lack of a Central line station at Shoreditch High Street. TOCs who are the main companies that interact with passengers and therefore would be the people who can see the potential for the future, were all massively undercapitalised - all the assets are owned by Network Rail or the rolling stock owning companies, who were quite happy sweating the assets rather than borrowing to produce future returns.
You might be able to fix that by twiddling around with the structure, but an attempt to integrate TOCs and the infrastructure would have to deal with the problem of "joint running" - where two TOCs share lines. Longer franchises have been mooted, but even 20 years isn't close to the length of a new train's expected lifetime.
So, given that the Overground is working, then why not expand that model? It's certainly a massively popular vote-winner in London, which is why even Boris Johnson wants to put as much of the London suburban network under TfL supervision as is practicable, and there's not even a doubt that Crossrail will be run like the Overground.
If we accept that some level of subsidy for the railways is an essential part of making this country work (as we do without question for the roads) then the question is how this money is best spent. Savings can be made by reducing the industry's internal bureaucracy (there are how many CEOs and directors all earning six-figure salaries?). Use the industry's true buying power by ordering large numbers of standard trains, which then can be deployed flexibly in accordance with changing demand rather than being tied to one TOC's pitch. Finish with this stop-start flow of orders, which adds to Derby's costs, and instead try to keep its order book full. And we can try to save money on the operational side of things by rolling out driver-only-operation everywhere (I'm sorry, but if it's safe on the tube, it's safe anywhere), while making guarantees that guards will still have jobs elsewhere in the railway.
With this money, let's displace journeys that would otherwise take place on the roads. This certainly involves lengthening trains and platforms, and perhaps reopening stations and lines to provide a more coherent railway network, and introducing suburban services where they have been withdrawn. Maybe that involves electrifying lines before we are forced to rush-electrify the entire network by peak oil 30 years from now. And perhaps we pick our battles, abandon the attempt to compete with air travel, and not build HS2, instead spending the sum on smaller projects with a greater collective impact.