Passivating contacts consisting of doped polycrystalline silicon (poly‐Si) on a thin tunnel‐oxide enable excellent operating voltages for crystalline silicon solar cells. However, hole‐collecting contacts based on boron‐doped poly‐Si do not yet reach their full surface‐passivation potential, likely due to boron diffusion during annealing. In this work, the authors show how the insertion of a thin intrinsic silicon buffer layer between the silicon oxide and poly‐Si is effective in improving the contact passivation. By tailoring the microstructure of the buffer layer, the chemical passivation and contact resistivity are simultaneously significantly improved. On device level, the buffer layer enables a ≈30 mV open‐circuit voltage enhancement and 1.4% absolute gain in power conversion efficiency.