In a National Journal article titled "The Volatile Senate," Ron Brownstein writes "Control of the Senate hasn't been this unsteady since the decades after the Civil War. It's no coincidence that was another period, like ours, of intense economic and social upheaval as America was reshaped by reinforcing waves of industrialization, immigration, and urbanization. And yet even then, the Senate wasn't unstable for as long as it has been since 1980."
Brownstein is correct that neither party has controlled the Senate for more than eight consecutive years in the last thirty-four years, whereas each party had long periods of dominance earlier in the twentieth century.
Yet, perhaps the most volatile period in terms of Senate control (although it lasted nowhere nowhere near as long as today's tumult) occurred during the 1950s. For four Congresses (the 82nd through 85th), neither party had more than a 2 seat margin. In fact, the 83rd Congress, which began with 48 Republicans, 47 Democrats, and 1 independent, featured the deaths of an unprecedented nine Senators, and many shifts of power.
According to Robert Caro's Master of the Senate, Democrats worried again in 1957 that they would lose control of the Senate when Senator Matt Neely died. This fear did not come to fruition because William Proxmire upset Walter Kohler in a Wisconsin Special Election, but the fear indicated how closely divided the parties were.
Why does this volatility matter?
Brownstein asserts that narrow margins inherently stifle compromise, and encourage partisan conflict (forthcoming work from political scientist Frances Lee similarly argues that "the increase in two-party competition for institutional control has incentivized congressional parties to organize themselves to pursue a more confrontational style."
Yet, I think the senates of the 1950s would call that premise into question. As documented by Caro, they were among the most productive senates of all time. Now some of that productivity was certainly a byproduct of Lyndon Johnson's brilliant tactical maneuvering.
Yet, the critical difference between those senates, and today's Senate, is that neither party was ideologically coherent in the 1950s. Liberal Republicans, such as Vermont's George Aiken and New Jersey's Clifford Case, and conservative Democrats, such as Georgia's Richard Russell and Mississippi's James Eastland, were frequent during the 1950s.
Today, even the number of moderates or pragmatic deal cutters in each party—let alone senators from the opposite ideological pole— is in the single digits.
Additionally, as Brownstein points out, most Republicans today are elected from "red" states won by their party's presidential nominee multiple elections in a row, and most Democrats hail from "blue" states. Brownstein argues that this may apply heightened pressure on these senators to reflexively back a president from their own party and oppose presidents from the opposite party.
I think that the issue has less to do with the president's party, and more to do with the electoral competitiveness in those states. Many of these senators must fear primary elections more than general elections. As a result, they have to worry most about being labeled insufficiently liberal or conservative.
With very few remaining ideological moderates (such as Maine's Susan Collins and West Virginia's Joe Manchin), our hope for a functional Senate must rely on less ideologically moderate senators like Lamar Alexander, John McCain, Ron Wyden, and Mark Warner, who still endeavor to legislate and address the nation's ills.
The press often mischaracterizes these senators as moderates. It is far more accurate to label them as mavericks, because while they aren't ideologically moderate, they are willing to buck the party line on occasion, and to engage in bipartisan negotiation on big issues.
Currently, these senators have been far too unwilling to assert themselves. The Republicans in the bunch are also not sufficiently willing to reject what Sean Theriault has dubbed "partisan warrior" behavior (gamesmanship and obstruction for the sake of obstructing). For the Senate to function, they must do so.
Additionally, the committee process must be allowed to work again. If we look at the great legislative compromises of years past (for example, the famous 1986 Tax Reform compromise), they were often the byproduct of chairmen working with their committees uninhibited by the party leadership.
Absent the parties changing their leadership, the willingness of senators looking to legislate to assert themselves and work around leadership will be critical to the Senate accomplishing anything.
Majority Leader Harry Reid and Minority Leader Mitch McConnell are both brawlers who are not overly popular back home and are, thus, used to winning by destroying the opposition. They are also both fantastic legislative tacticians, who have mastered the procedural rules, and used them to their party's advantage. These skills are far better suited to pursuing a partisan agenda than to compromising in order to accomplish much needed legislation.
Additionally, acrimony between Reid and McConnell makes it more difficult for them to lead the Senate towards greater productivity. Both leaders also face substantial pressure from their bases that makes it difficult to resist doing things that hurt the institution, but benefit either their own careers, or their respective parties (at least in the short term).
Thus, the mavericks must lose their reticence to challenge the leadership, and work around them in order to address issues like our crumbling infrastructure, the danger of a student loan bubble, etc. This technique might not be successful on some valence issues (for example, healthcare— where the system still needs major fixes). But it should be able to bridge philosophical divides on less partisan issues.
This is not to argue that they will always be successful. Indeed, much else has changed since the senates of the 1950s. A vastly improved system of air travel has truncated the senate work week, and reduced the bonds between senators who built friendships while living with their families in Washington, DC. Voters have sorted themselves into more coherent parties and/or polarized. The filibuster is no longer a tactic held back for the most extreme situations (and that genie likely can't be put back in the bottle). New media and social media have enabled even the most junior senator to ignore the whims of leadership and become political forces.
But it is the best hope that we have for a functional legislative process moving forward.