In their CNN town halls Monday night, Democratic presidential candidates Sen. Bernie Sanders and Mayor Pete Buttigieg disagreed on whether current prisoners should be able to vote. Sen. Kamala Harris refused to endorse a plan for expanding the franchise to incarcerated people, but supported voting rights for former prisoners.

Sanders was specifically asked about Boston Marathon bomber Dzhokhar Tsarnaev and “those convicted of sexual assault.” What sane person would want them to vote? Our political system is already run by crooks. Do we want to add murderers and rapists too?

In European history dating to Roman times, criminals could be stripped of their legal personality after committing a crime. They could not sign contracts or own property. They were outlaws, banished from the city walls. John Locke and other political theorists argued that criminals broke an implicit social contract: a rule-breaker should lose the right to make rules for others.

But Locke lived in a time when only white, male, wealthy landowners could vote. Today, the right to vote is enshrined in democratic constitutions and international treaties. In American history, many states’ exclusions of those with a criminal record from voting date to the post-Civil War period and were clearly aimed at denying the franchise to African Americans.

Criminal justice reform advocates argue that suffering a Medieval-style “civil death” dehumanizes prisoners, prevents their reintegration into society, and perpetuates inequalities in our political system. We should not assume that prisoners are less knowledgeable about politics than those outside of prison—that’s a pretty low bar, after all. Encouraging prisoners to feel involved in the political process can have real benefits too. Isolating prisoners from the political process during and after their incarceration further stigmatizes and isolates them, and that can encourage reoffending.

Prisoners lose many of their rights when they go to prison. They can’t serve on a jury from a prison cell, or own guns; both of those are probably reasonable proscriptions. They probably should not own guns. But prisoners do not lose all their rights in prison. They are entitled to practice their religion and can challenge the conditions of their confinement. Taking away prisoners’ liberty is already a heavy punishment. Allowing them to cast an absentee ballot is not an unreasonable privilege.

The most important consequence of allowing prisoners to vote is that it would remove the incentives for “prison gerrymandering.” In most U.S. states, prisoners are counted by the census based on where they are incarcerated, not where they are registered to vote. Because most large prisons are in sparsely populated rural areas, prison complexes have an important effect on gerrymandering.

Many prisoners are racial minorities or people who live in urban areas, which means these places lose voting population, while more conservative areas gain nonvoting population. This advantages Republican congressmen in places like upstate New York, who benefit from inflated populations for redistricting purposes, but have nothing to fear at election time. Prisoner disenfranchisement therefore contributes to a structural disparity that causes Congress and state legislatures to be more conservative than the public at large.

While many states are in the process of revising their laws to allow ex-prisoners to vote, voting by current prisoners only exists in Maine, Puerto Rico, and Vermont—the latter represented by Sanders in the U.S. Senate. In addition, the trend across the developed world is to allow at least some prisoners to vote. The supreme courts of South Africa, Canada, and Israel have legalized voting for at least some prisoners. The European Court of Human Rights has also rejected blanket prohibitions on prisoner voting, though it has allowed exceptions.

The policy options are far broader than a single audience question would suggest. In Germany, prisoners can vote unless they were convicted of terrorism or political violence, an exception that would encompass Tsarnaev’s marathon attack. Other European countries prevent violent criminals, those serving lengthy or life sentences, or war criminals from voting. Exceptions for crimes of dishonesty or fraud might be reasonable as well. In a few countries, only those convicted of misdemeanors can vote, rather than felonies.

These are policy debates we should be willing to have. Even if we allowed only persons serving misdemeanor sentences in local jails to vote, this alone might add nearly 300,000 voters to the rolls.

Prisoner voting is already underway in some states and developed countries, so it is hardly a revolutionary position. Overbroad restrictions on voting help ensure that politicians select their own voters, rather than voters electing their own politicians.

Andrew Novak is Assistant Professor of Criminology Law and Society at George Mason University.