We investigate computation through the lens of dynamical systems, unifying physical processes and machine learning. By treating both hardware and algorithms as evolving dynamical systems, we leverage natural physical dynamics - such as relaxation to stable states and phase transitions - as a foundation for robust, efficient computation. Our work shows that concepts from physics - like order parameters and criticality in the Ising model - directly shape learned representations in self-supervised auto-encoders, revealing deep connections between physical phase transitions and optimization dynamics in neural networks. By framing computation as constrained dynamical evolution, we enable abstraction across scales: algorithms emerge predictably from physical behavior, decoupled from hardware specifics. This approach, grounded in dynamical systems theory, paves the way for general-purpose physical computers that are energy-efficient, scalable, and inherently adaptive.