Switzerland has rejected a cap on the country's population when the proposal was put to the vote in a nationwide referendum.