The capacity to faithfully perceive, store and recall sequential information lies at the heart of human cognition and ability for language and culture. Other animals lack such precision and seem instead to represent sequences of stimuli as unstructured collections of memory traces. Therefore, an inborn ability to perceive and use sequential information may be a necessary step in human evolution that predated other uniquely human abilities. We suggest a theoretical basis for this hypothesis and formalize a tentative first step that may have taken place in the evolution of the human brain. With mathematical modelling and computational simulations, we explore under what circumstances evolution would favour different kinds of memory adaptations, and identify costs and benefits with sequence memory. A central question is why this capacity has only evolved in one species.