Advancing non-fullerene acceptor (NFA) organic photovoltaics requires the mitigation of the efficiency-limiting processes. Acceptor end-group and side-chain engineering are two handles to tune properties, and a better understanding of their specific impact on the photophysics could facilitate a more guided acceptor design. Here, the device performance, energetic landscape, and photophysics of rhodanine and dicyanovinyl end-capped IDT-based NFAs, namely, O-IDTBR and O-IDTBCN, in PCE10-based solar cells are compared by transient optical and electro-optical spectroscopy techniques and density functional theory calculations. It is revealed how the acceptors’ quadrupole moments affect the interfacial energetic landscape, in turn causing differences in exciton quenching, charge dissociation efficiencies, and geminate versus non-geminate recombination losses. More precisely, it is found that the open circuit voltage (VOC) is controlled by the acceptors’ electron affinity (EA), while geminate and non-geminate recombination, and the field dependence of charge generation, rely on the acceptors’ quadrupole moments. The kinetic parameters and yields of all processes are determined, and it is demonstrated that they can reproduce the performance differences of the devices’ current–voltage characteristics in carrier drift-diffusion simulations. The results provide insight into the impact of the energetic landscape, specifically the role of the quadrupole moment of the acceptor, beyond trivial considerations of the donor–acceptor energy offsets.