Significant efforts are being made worldwide to understand the immune response to SARS-CoV-2, responsible for the COVID-19 pandemic, including the role of preexisting T cell immunity. Understanding the mechanisms that promote cross-recognition by T cells induced by seasonal coronaviruses will be critical for future predictions on the role of pre-existing immunity in protection against severe disease. We demonstrate that the SARS-CoV-2 nucleocapsid (N) protein induces an immunodominant response in HLA-B7+ COVID-19-recovered individuals that is also readily detectable in unexposed donors. This immunodominant response is driven by a single N-encoded epitope that displays a high degree of conservation with the homologous region in circulating coronaviruses. We show that T cell-mediated cross-reactivity can be detected towards the circulating OC43/HKU-1 coronaviruses, but not the 229E or NL63 coronaviruses, due to different peptide conformations. This cross-reactivity is driven by private T cell receptor repertoires with a bias for TRBV27 and a long CDR3b loop in unexposed and COVID-19-recovered individuals. Together, our findings demonstrate the basis of pre-existing immunity to a conserved and highly immunogenic SARS-CoV-2 epitope driven by cross-reactive memory T cells, suggesting long-lived protective immunity.