Understanding how the brain transforms sensory input into complex behavior is a fundamental question in systems neuroscience. Using larval zebrafish, we study the temporal component of phototaxis, which is defined as orientation decisions based on comparisons of light intensity at successive moments in time. We developed a novel “Virtual Circle” assay where whole-field illumination is abruptly turned off when the fish swims out of a virtually defined circular border, and turned on again when it returns into the circle. The animal receives no direct spatial cues and experiences only whole-field temporal light changes. Remarkably, the fish spends most of its time within the invisible virtual border. Behavioral analyses of swim bouts in relation to light transitions were used to develop four discrete temporal algorithms that transform the binary visual input (uniform light/uniform darkness) into the observed spatial behavior. In these algorithms, the turning angle is dependent on the behavioral history immediately preceding individual turning events. Computer simulations show that the algorithms recapture most of the swim statistics of real fish. We discovered that turning properties in larval zebrafish are distinctly modulated by temporal step functions in light intensity in combination with the specific motor history preceding these turns. Several aspects of the behavior suggest memory usage of up to 10 swim bouts (~10 sec). Thus, we show that a complex behavior like spatial navigation can emerge from a small number of relatively simple behavioral algorithms.