Since the 2000 U.S. Presidential election, social scientists have rediscovered a long tradition of research that investigates the effects of ballot format on voting. Using a new dataset collected by the New York Times, we investigate the causal effect of being listed on the first ballot page in the 2003 California gubernatorial recall election. California law mandates a unique randomization procedure of ballot order that, when appropriately modeled, can be used to approximate a classical randomized experiment in a real world setting. We apply (nonparametric) randomization inference based on Fisher's exact test, which directly incorporates the actual randomization procedure and yields accurate confidence intervals. Our results suggest that over forty percent of the minor candidates gained more votes when listed on the first page of the ballot, while there is no significant effect for top two candidates. We also investigate how randomization inference differs from conventional estimators that do not fully incorporate California's complex treatment assignment mechanism. The results indicate appreciable differences between the two approaches.