the hidden political party

the hidden political party 2022-06-20 · 4 min read a cluster analysis of tweet sentiments to find sub-political parties — hopefully this helps people understand the political landscape a little more POLITICS! it's a crazy time of the decade where it seems like everyone is disagreeing about something (rightfully so in some cases). in light of all these differences, it still amazes me how we manage to split an entire country into only two parties with seemingly infinite sides to infinite problems. in reality there should be ideologically many more parties, so i set out to see if i could derive any insights from this mathematically. the process first i started off with a public dataset of most major US politicians' twitter accounts. once i consolidated all the usernames, i scraped all their current tweets up to around 300 per account. taking mentions into account, i built a graph where every person was connected with the people they had mentioned in their tweets. this on its own gives you a graph that shows the "relatedness" of different politicians — but the problem is it doesn't tell you if the relationship is positive or negative. two people could be bitter rivals or political best friends and the graph would look the same. to fix this i took sentiment of the tweets into account, letting that control the weightings of the edges — essentially creating a weighted undirected graph as a representation of the current political state of the US. it's not going to be perfectly accurate to the complexity of real political relationships, but it acts as a good approximation of outward personas. with that in place, i was able to run clustering algorithms! specifically the Louvain Heuristic, which works well on large datasets with many connections. i let it classify itself and see what came out. the insights okay so onto the slightly clickbait-y title. one of the most surprising results — or least surprising depending on how you look at it — was that the algorithm didn't identify just 2 political groups. it found 3. looking into the groupings, it sorted all the politicians into what i'm calling Left Democrats, Conservative Democrats, and Republicans. the model found that the left wing was genuinely divided into two distinct clusters — one actively pushing for more progressive legislation, the other staying closer to the status quo. Republicans on the other hand sorted cleanly into one unified group with relatively few cross-cluster connections. [graph: cluster analysis of US politicians by tweet sentiment] two big possible explanations here. either the Democratic party is just more openly debating issues because they're in the presidential term, or a new more left-leaning party is genuinely emerging from within. the former seems less likely to me given roughly equal political control at the time — which leaves the more interesting conclusion: there's a real ideological split happening inside the Democratic party that the two-party system is papering over. this splitting is something most people already sense. what's interesting is seeing it show up mathematically without any human labeling. the graph just found it. one more thing worth noting — Left Democrats get significantly more mentions and media attention than Conservative Democrats despite being similar in number. which raises an interesting question about whether a formal party division would actually benefit Conservative Democrats, who currently seem to be getting drowned out inside their own coalition. so what i'm not going to claim this is a perfect model of US politics. it's a sentiment-weighted graph of tweets, not a comprehensive theory of political science. but i do think it's a useful approximation — and the fact that it independently surfaced a three-party structure that most people already intuitively feel is at least worth thinking about. it would be cool to see more formal debates between both types of Democrats about the actual direction of the party, rather than the whole thing being framed primarily as anti-Republican. the data suggests they're two different things that happen to share a party label. and..... overall this was a really fun side project and definitely something i want to keep doing! applying data science techniques to real-world problems and actually getting insights out of it is exciting. i've linked the graph visualization so you can take a look and hopefully find something interesting yourself. see you soon, and keep hacking!