What Madison was Afraid of