diff --git a/README.md b/README.md index 6fb2b37..198127e 100644 --- a/README.md +++ b/README.md @@ -4,12 +4,17 @@ A whole-brain *Drosophila melanogaster* connectome simulation that plays Five Nights at Freddy's 1. +![A night 3 played by the fly, the game on the left and its brain on the right](docs/media/night-3.gif) + +Night 3 from midnight to 5 AM, sped up. The game is on the left and the fly's brain is on the right, every dot a neuron lit as it fires. + 138,639 of the 139,255 proofread neurons in the FlyWire 783 connectome run as leaky integrate-and-fire units, wired by the measured synaptic weights. That is 99.56% of the brain, and the ones left out are almost all sensory afferents rather than interneurons. Nothing else is subset. The screen is fed into the fly's visual clusters, and the mouse is driven by reading its descending neurons. ## Best run so far -The fly reached 4 AM on night 2. The run is recorded at -https://www.youtube.com/watch?v=4UNPlA-YJtw. +The fly reached 5 AM on night 3 with 9% of the power left, and Foxy got in +before 6. That is the night in the clip above, and the video is at +https://www.youtube.com/watch?v=R4Ogbf33Gfo. ## What the fly actually does @@ -17,13 +22,21 @@ Four behaviours come out of the network, and each one is a different pathway. **Slamming a door.** The hallway is captured while the light is on, which the fly confirms by reading the light button on screen, and compared against the views of that hallway recorded empty at midnight, lit, flickering and dark. A frame that is far from all of them drives the corresponding eye cluster at full rate, every LPLC2 and LC4 neuron on that side, that excitation propagates through the real connectivity, and when DNp01, the Giant Fiber, crosses its own firing threshold the door closes. Measured on this machine, the neuron needs seven to nine engine frames of sustained input to answer, which is why the light is held for twelve. The door stays shut while the fly can still see the threat. With the door closed the light still reaches the window, where Bonnie casts a shadow and Chica stands, and the eye reads that window on its own against the lit view of the empty window behind a closed door, so a dark window never passes for an empty one. A threat still there fires DNp01 again and renews the hold, and the door only reopens once the defensive state has decayed and the latest look at that side, taken with the light confirmed on, came back empty (ADR 0011, experiment 14). +![Chica at the right window, DNp01 fires and the right door closes](docs/media/door-on-chica.gif) + +Chica is at the right window. The panel reports DNp01 on the right firing, traces the spike back to the eye cluster, and the door comes down. + **Checking a hallway.** The difference in membrane potential between the left and right eye populations is a slow spontaneous signal the network produces on its own, with a lag-1 autocorrelation of 0.97. It is accumulated to a bound together with what the last look revealed and a per-side habituation term, and whichever side wins gets looked at. There is one starvation guard, a clock that forces a look at a hallway left unwatched for thirty seconds, and every look it causes is counted separately in the session report so its share stays visible. **Raising the monitor.** DNp09's own subthreshold membrane potential is accumulated the same way. A real spike raises the tablet outright, and the tablet comes down when that drive fades rather than when a timer expires. Raising the tablet also drives the GABAergic inhibitor clusters at full rate, which silences the Giant Fiber, so no door reflex can fire through the tablet, exactly as no door button can be pressed in the game. Whether the tablet is up is read off the screen, by counting the eleven camera buttons of the map, and the fly believes a raise or a lowering only once the map has appeared or gone (ADR 0015). -**Watching for Foxy.** The tablet always comes up on Pirate Cove, CAM 1C. What differs from the cove as it looked at the start of the night drives LC9 and LC31a, the lobula columns that feed DNp09 and are used by flies to pursue a moving object. When DNp09 fires, the gaze follows to the west hall, CAM 2A. There the size of whatever differs drives LPLC2 and its growth drives LC4, and DNp04, a looming escape neuron the inhibitors barely reach, fires on it. The fly then drops the tablet and closes the left door. Measured on the engine, each channel wakes its own descending neuron and never the other, and the Giant Fiber stays silent through both (ADR 0026, experiment 13). +**Watching for Foxy.** The tablet always comes up on Pirate Cove, CAM 1C. What differs from the cove as it looked at the start of the night drives LC9 and LC31a, the lobula columns that feed DNp09 and are used by flies to pursue a moving object. When DNp09 fires on the cove, the fly keeps a memory of the object on its left that fades over about ten seconds, the way a fly keeps heading for a landmark that has vanished (Neuser et al. 2008). That memory drives LPLC2 and LC4 on the left, and DNp04, a looming escape neuron the inhibitors barely reach, fires on it. The fly then drops the tablet and closes the left door, and with the tablet down the same memory keeps the Giant Fiber firing, which holds the door for about twenty seconds. The west hall is never looked at, because Foxy reaches the door 1.7 seconds after CAM 2A shows him and the escape takes longer than that. Measured on the engine, each channel wakes its own descending neuron and never the other, and the Giant Fiber stays silent while the tablet is up (ADR 0026, experiment 13). + +![Foxy peeking out of Pirate Cove, the tablet drops and the left door closes](docs/media/foxy-from-the-cove.gif) + +Foxy is peeking out of the curtain on CAM 1C. The fly drops the tablet and the left door is shut before he runs. ## What this is not @@ -141,9 +154,9 @@ each region actually reads. Each one runs as a module, for example - `src/flynaf/scripts/debug_vision.py` shows live what each eye region is reading, and `src/flynaf/scripts/debug_camera.py` prints how many camera buttons the map region holds and whether that counts as the tablet being up. - `src/flynaf/scripts/debug_reflex.py` shows the live contrast of both hallways against the threshold, and says whether a door would slam. Run it with a light held on, because a reading taken with the light off means nothing. - `src/flynaf/scripts/measure_door_window.py`, given `left` or `right`, closes that door, lights the window and prints how far it strays from the closed-door reference. Run it once with the hallway empty and once with Bonnie or Chica outside, and set `closed_door_mse_threshold` between the two readings. -- `src/flynaf/scripts/debug_tablet.py`, given `1C` or `2A`, records one camera with nothing in view and then prints the live drive it would send into the brain. That is how the spans in `TabletVision` are fitted, and how the CAM 2A button position, inferred from CAM 4B, is confirmed. +- `src/flynaf/scripts/debug_tablet.py`, given `1C`, records the camera with nothing in view and then prints the live drive it would send into the brain. That is how the span in `TabletVision` is fitted. -The camera references are not cached. At the start of every night the fly raises the tablet once, records a few seconds of CAM 1C and CAM 2A while the curtain is closed and the hall is empty, and lowers it again, which also stalls Foxy while it happens. +The camera references are not cached. At the start of every night the fly raises the tablet once, records a few seconds of CAM 1C while the curtain is closed, and lowers it again, which also stalls Foxy while it happens. The eye references are captured once and cached in `logs/vision_reference/`, alongside a fingerprint of the layout they were taken on. They are re-measured @@ -313,3 +326,4 @@ Five Nights at Freddy's is by Scott Cawthon and is not affiliated with this proj | 1.17 | Moved the paths and commands to the flynaf package, described the pytest and CI pipeline and the connectome marker | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-01 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-01 | | 1.18 | Described the door that reopens after a clearing look, the closed-door calibration, the tablet read from the camera map, the warm-up and the bank of hallway views | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-01 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-01 | | 1.19 | Described the window read behind a closed door, the lit look a door needs to reopen and the game moved aside for the panel | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-01 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-01 | +| 1.20 | Added three clips of a night 3, the new best run, and the memory of Foxy that replaced the look at CAM 2A | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-06 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-06 | diff --git a/docs/adr/0026-the-tablet-feeds-the-lobula-and-dnp04-answers-through-it.md b/docs/adr/0026-the-tablet-feeds-the-lobula-and-dnp04-answers-through-it.md index ad440c4..d9e4f2a 100644 --- a/docs/adr/0026-the-tablet-feeds-the-lobula-and-dnp04-answers-through-it.md +++ b/docs/adr/0026-the-tablet-feeds-the-lobula-and-dnp04-answers-through-it.md @@ -20,28 +20,55 @@ figure 0.25 to 1.00 0/6 0/6 6/6 @ 3 ``` ## Decision -The tablet reads the west side through two lobula channels, chosen by the camera being watched. +The tablet reads Pirate Cove through LC9 and LC31a, and what DNp09 answers to is remembered on the left through LPLC2 and LC4. - Raising the tablet taps CAM 1C. This is a fixed motor gesture, recorded as such, in the same class as the pan delay. -- The cove is compared against a bank of frames captured at the start of the night with the curtain closed. Taking the nearest frame in the bank absorbs the camera pan. The distance above the bank's own noise, over `figure_span_mse`, drives LC9 and LC31a on the left. -- A DNp09 spike while the cove is readable moves the gaze to CAM 2A, once per raise. The object has left and the fly follows it. -- In CAM 2A the same distance drives LPLC2 and its growth per second drives LC4, the size and velocity split of the door reflex. +- The cove is compared against a bank of frames captured at the start of the night with the curtain closed. The feed is the whole camera view with the map, its label, the clock and the power readout masked out, because the gap in the curtain, the sign and the far end of the west hall all sit outside the 360 pixel square the feed first read. Each frame is slid sideways against its nearest references before they are compared, since the camera pans about 160 pixels each way and a bank recorded over three seconds covers only part of that sweep. The distance is the mean of the three strongest cells of 40 by 40 pixels rather than the mean over the frame, the way a columnar lobula population reports a small object wherever it is instead of averaging it away. That distance above the bank's own noise, over `figure_span_mse`, drives LC9 and LC31a on the left. +- A DNp09 spike while the cove is readable stores an object memory on the left, as strong as the figure drive at that moment, fading with `object_memory_sec`. Flies keep heading for a landmark for seconds after it vanishes (Neuser et al. 2008) and stay in a defensive state for tens of seconds after a visual threat (Gibson et al. 2015). A spontaneous DNp09 spike on a closed curtain stores nothing, because the figure drive is zero. A weaker sighting never overwrites a stronger memory, and a memory that would feed less than one spike a second per neuron counts as gone. +- The memory drives LPLC2 and LC4 on the left at its current level, whether the tablet is up or down. No rule says to close the door. With the tablet up DNp04 answers, and with the tablet down the Giant Fiber answers, which renews the door hold of ADR 0015. +- CAM 2A is not looked at. Until version 1.3 a DNp09 spike moved the gaze there and the hall drove LPLC2 and LC4. Foxy reaches the door 1.7 seconds after CAM 2A first shows him, and reading the hall, firing DNp04, lowering the tablet and closing the door do not fit in that. Left unwatched, he takes 25 seconds from leaving the cove. The hall also could not be read. Its lamp flickers, the camera pans further than the bank covers, and the map covers most of his run, so on a recorded night 3 the empty hall and Foxy running gave overlapping distances under every metric tried. - A DNp04 spike while the tablet is up drops the tablet, waits for the camera map to leave the screen and only then closes the door on that side (ADR 0015). - A watch only counts as boring once the feed has been readable for `camera_watch_min_sec`. Counted from the raise, as it first was, every watch on 2026-10-01 ended 0.7 to 1.0 seconds in, before the 1.2 second settle had passed, and CAM 1C never reached the brain once in a night Foxy won. Counted from the moment the feed is readable, the same night's watches fed 0.02 to 0.27 of figure drive while Foxy changed in the cove. That sits around the 0.25 the table above needs for DNp09, so a peeking Foxy is noticed weakly and rarely pursued, and the empty cove with its sign has not yet been caught on a watch. +- On 2026-10-02 a night 2 was played by hand while every CAM 1C and CAM 2A frame was saved, and the old square read the empty cove at 0.04 of figure drive. Replayed through the reading above, against nine closed-curtain frames from the same night, the stages separate cleanly, and `figure_span_mse` moved from 1000 to 1500 so the empty cove lands near 0.6. + +``` +CAM 1C, night 2 contrast figure drive +closed curtain (noise) 2 to 278 0 +Foxy peeking 530 to 1335 0.14 to 0.67, median 0.42 +Foxy out of the curtain 788 to 1549 0.31 to 0.82, median 0.58 +empty cove with the sign 1172, 1192 0.57, 0.58 +``` + +- A night 3 recorded on 2026-10-03 read a closed curtain at contrast near 1000 on some watches. The MUTE CALL button and the blinking recording dot sit at the top left of the view and were outside the mask, and CAM 1C pans about 320 pixels in total, more than the 200 the alignment could slide. The top left mask grew to cover both and `pan_reach` moved to 360, after which the closed curtain of that night reads at the bank's noise. +- With the tablet down and the same loom drive, DNp01 answered 6/6 from 0.02 in trials started from rest. In a continuous run the network adapts and the Giant Fiber stops answering once the memory falls to about 0.1. Four runs per strength, tablet up for the first 1.5 seconds, gave the figures below. `object_memory_sec` is 10 because Foxy needs 25 seconds from the cove to the door and the camera refractory brings the tablet back to the cove after 15, where a cove that is still empty refreshes the memory. + +``` +memory DNp04 fires last Giant Fiber spike door hold ends +4 s 0.2 s 5.3 to 9.0 s 10.1 to 13.8 s +8 s 0.2 s 10.3 to 16.2 s 15.1 to 21.0 s +10 s 0.2 s 13.2 to 20.9 s 18.0 to 25.7 s +``` + +- A night run on 2026-10-04 watched CAM 1C ten times and read 0.37 to 1.00 of figure drive on nine of them, yet stored one memory. The inhibitors hyperpolarise DNp09 during the 1.2 second settle, so after it the neuron needs 6 to 7 engine frames to answer a figure of 0.4, 5 at 0.6 and 4 at 1.0, eight trials each, which is 0.8 to 1.2 seconds on that night's frames. `camera_watch_min_sec` was 0.8, so on four watches DNp09 spiked one or two frames after the fly had already given up the watch, with nothing left to remember. The last of them read 1.00 about seventeen seconds before the night ended. `camera_watch_min_sec` is now 2.0, which also covers most answers to a figure of 0.2 (9 to 17 frames). +- The camera map can take longer than `flip_confirm_sec` to appear on the very first raise of a night. Five nights that started with the eye references loaded from disk raised the tablet about 0.1 seconds after the office appeared, missed the map and found the tablet up a few seconds later, so those nights never recorded a bank and the tablet fed nothing at all. The start-of-night raise now waits up to `start_raise_patience_sec` for the map before giving up, and logs how late it came. It never repeats the gesture, because a second flip during a slow raise would put the tablet back down. `env/tablet_feed.py` captures and compares the feed, `night/tablet/` holds the transduction, the gaze, the per-frame watch and the start-of-night references, and the door bookkeeping moves out of the engine loop into `night/doors.py` so the Giant Fiber and DNp04 close doors through one path. ## Consequences -- Which descending neuron answers is decided by the connectome. Neither channel reaches the other's readout, and the Giant Fiber stays gated throughout, so `test_inhibition.py` keeps its guarantee and now also asserts the dissociation. +- Which descending neuron answers is decided by the connectome. Neither channel reaches the other's readout, and the Giant Fiber stays gated while the tablet is up, so `test_inhibition.py` keeps its guarantee, asserts the dissociation, and asserts that a remembered figure keeps the Giant Fiber answering once the tablet is down. - Seeing Foxy in the cove keeps DNp09 depolarised, which keeps the tablet up longer, which stalls him. That behaviour is not written anywhere, it follows from DNp09 already controlling the tablet. -- Bonnie standing in 2A drives LPLC2 without growth and still wakes DNp04, so the left door closes early. That is a correct defensive reading and costs power. -- The spans that convert image distance into drive are game-side constants that have not been fitted against a recorded Foxy. `debug_tablet.py` prints them live, and traces now carry the tablet drive so they can be refitted from recorded nights. -- The CAM 2A button position is inferred from CAM 4B and needs confirming on screen. -- The start of the night gains one tablet raise of about eight seconds to record the two banks. Foxy is stalled for all of it. -- The escape is a chain of motor actions, and lowering the tablet plus panning to the left door takes over a second. Whether that fits inside Foxy's run is the first thing a recorded night has to answer. +- Foxy peeking through the curtain reads about 0.42, enough for DNp09 and for a memory, so the left door can close for about twenty seconds while he is still in the cove. That costs power and has not been measured on a full night. +- A raise that comes while the memory is still above about 0.15 makes DNp04 fire again and drops the tablet before the cove is read. The door is closed at that point, and the next raise reads the cove. +- The figure span is fitted on one night of Foxy in the cove. `debug_tablet.py` prints the level live, and traces carry the tablet drive so it can be refitted from recorded nights. +- Reading the whole view costs about 50 milliseconds a frame with the wider pan reach, so the feed computes it on its own capture thread and the vision loop only collects the latest value. The in-game panel can no longer sit over the camera view, since it would cover the gap in the curtain. A game that leaves no room beside it now gets the panel elsewhere on the screen or none (ADR 0009). +- The start of the night gains one tablet raise to record the CAM 1C bank. Foxy is stalled for all of it. +- The escape is a chain of motor actions, and lowering the tablet plus panning to the left door takes over a second. It now starts when the cove is seen changed, with up to 25 seconds in hand. The whole chain is tested on the engine and on recorded frames. The one live night so far stored a single memory, which closed the left door four seconds later through the Giant Fiber and held it 22.5 seconds, and a live night with the longer watch is still owed. | Version | Description | Author(s) | Date | Reviewer(s) | Review Date | |---|---|---|---|---|---| | 1.0 | CAM 1C drives LC9 and LC31a, DNp09 pursues to CAM 2A, CAM 2A drives LPLC2 and LC4, DNp04 closes the door | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-09-30 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-09-30 | | 1.1 | Described the escape against the camera map instead of the office reference | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-01 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-01 | | 1.2 | Counted the minimum watch from the moment the feed is readable and recorded the drive Foxy produced | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-01 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-01 | +| 1.3 | Read the whole camera view, aligned to the pan and measured by its strongest cells, refitted the figure span on a recorded night 2 and waited for a slow first raise | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-02 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-02 | +| 1.4 | Dropped the CAM 2A pursuit. A DNp09 spike on the cove stores an object memory that drives LPLC2 and LC4, so DNp04 closes the door and the Giant Fiber holds it. Masked the MUTE CALL corner and widened the pan reach | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-04 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-04 | +| 1.5 | Raised the minimum watch to 2.0 seconds after a live night where DNp09 answered the cove just after the watch had ended | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-04 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-04 | diff --git a/docs/experiments/13-foxy-was-never-on-the-screen-it-watched.md b/docs/experiments/13-foxy-was-never-on-the-screen-it-watched.md index 1f5b6c3..2290d7e 100644 --- a/docs/experiments/13-foxy-was-never-on-the-screen-it-watched.md +++ b/docs/experiments/13-foxy-was-never-on-the-screen-it-watched.md @@ -82,14 +82,19 @@ tenth and a quarter of the span on both channels. ## What The Fly Does Now -The tablet always comes up on Pirate Cove. The cove is compared against a bank -of frames recorded at the start of the night, while the curtain is still -closed, and the distance drives LC9 and LC31a on the left. When DNp09 fires -the gaze follows the object down the west hall to 2A, once per raise. In 2A -the distance drives LPLC2 and its growth drives LC4, and when DNp04 fires the -fly drops the tablet, closes the left door, comes back to centre and checks the -office before it trusts the inhibitors are off again. Which camera to open -first is a fixed gesture and I would rather say so than dress it up. +The tablet always comes up on Pirate Cove, and it is the only camera the fly +watches. The cove is compared against a bank of frames recorded at the start +of the night, while the curtain is still closed, and the distance drives LC9 +and LC31a on the left. When DNp09 fires on a cove the fly can read, it keeps a +memory of the object on its left that fades over about ten seconds. The memory +drives LPLC2 and LC4 on that side, and when DNp04 fires the fly drops the +tablet, closes the left door, comes back to centre and checks the office +before it trusts the inhibitors are off again. With the tablet down the same +memory keeps the giant fiber firing, which holds the door for about twenty +seconds. The west hall is left out, because Foxy reaches the door 1.7 seconds +after CAM 2A shows him and the escape takes longer than that (ADR 0026). Which +camera to open first is a fixed gesture and I would rather say so than dress +it up. Any camera already stalls Foxy, so the fly watching the cove longer when something is in it is the strategy a good player uses, and here it falls out @@ -97,18 +102,24 @@ of a neuron that was already doing the watching. ## What Is Not Measured Yet -Everything on the game side. The spans that turn an image distance into a -drive level, 1000 for both channels and 3000 per second for growth, are placed -so a clearly visible Foxy lands well above a quarter of the span, but I have -not recorded him. The 2A button position comes from the symmetry with 4B and -has to be checked. `src/scripts/debug_tablet.py` prints the live levels for -one camera so the spans can be fitted, and every trace now records the tablet -drive frame by frame, so a night with Foxy in it can be replayed through the -brain the same way the hallways are. - -The part I trust least is time. The run from 2A to the office is short, and -lowering the tablet, panning to the door and pressing it takes most of it. +Most of the game side. The one live night with Foxy in it, on 2026-10-04, read +the cove ten times. The first read nothing behind the closed curtain and the +other nine landed between 0.37 and 1.00 of the span, so a peeking Foxy already +sits above the quarter the brain asks for. Only one of those nine became a +memory, because the fly gave up the watch 0.8 seconds after the feed turned +readable and DNp09 needs four to seven engine frames to answer. The minimum +watch is now 2.0 seconds and a night with it is still owed. +`src/scripts/debug_tablet.py` prints the live levels for one camera, and every +trace records the tablet drive frame by frame, so that night can be replayed +through the brain the same way the hallways are. + +The part I trust least is time. Foxy arrives 25 seconds after he leaves the +cove and the memory holds the door for 18 to 26, so the door can reopen just +before he gets there unless the fly goes back to the cove and reads it empty +again. What every closing costs in power over a whole night is not measured +either. | Version | Description | Author(s) | Date | Reviewer(s) | Review Date | |---|---|---|---|---|---| | 1.0 | Why the fly never saw Foxy, the cell types behind the fix and the double dissociation that justifies it | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-09-30 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-09-30 | +| 1.1 | The fly no longer follows Foxy to CAM 2A, and the first live night with him is reported | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-05 | [Artur Mendonça Arruda](https://github.com/ArtyMend07) | 2026-10-05 | diff --git a/docs/media/door-on-chica.gif b/docs/media/door-on-chica.gif new file mode 100644 index 0000000..97f4a25 Binary files /dev/null and b/docs/media/door-on-chica.gif differ diff --git a/docs/media/foxy-from-the-cove.gif b/docs/media/foxy-from-the-cove.gif new file mode 100644 index 0000000..9d33c47 Binary files /dev/null and b/docs/media/foxy-from-the-cove.gif differ diff --git a/docs/media/night-3.gif b/docs/media/night-3.gif new file mode 100644 index 0000000..7724eeb Binary files /dev/null and b/docs/media/night-3.gif differ diff --git a/src/flynaf/config.py b/src/flynaf/config.py index 95a80f8..3ed4415 100644 --- a/src/flynaf/config.py +++ b/src/flynaf/config.py @@ -65,7 +65,7 @@ class ForagingParams: camera_refractory_sec: float = 15.0 saccade_refractory_sec: float = 2.5 camera_watch_max_sec: float = 6.0 - camera_watch_min_sec: float = 0.8 + camera_watch_min_sec: float = 2.0 camera_release_forage_bias: float = 0.5 closed_door_mse_threshold: float = 30.0 @@ -191,16 +191,24 @@ class CameraButton: x: int y: int side: str - channel: str - pursue_to: str | None = None @dataclass(frozen=True) class TabletVision: - feed_x: int = 470 - feed_y: int = 330 - feed_size: int = 360 - feed_pixels: int = 64 + view: tuple = (40, 20, 1250, 640) + hud: tuple = ( + (820, 280, 1250, 640), + (1100, 20, 1250, 110), + (40, 20, 160, 110), + (40, 600, 300, 640), + ) + view_scale: int = 8 + pan_reach: int = 360 + pan_coarse_step: int = 3 + nearest_references: int = 3 + cell_pixels: int = 5 + strongest_cells: int = 3 + start_raise_patience_sec: float = 6.0 capture_delay_sec: float = 0.03 map_ready_sec: float = 0.4 raise_settle_sec: float = 1.2 @@ -208,15 +216,12 @@ class TabletVision: reference_sweep_sec: float = 3.0 reference_frames: int = 24 noise_margin: float = 1.5 - figure_span_mse: float = 1000.0 - loom_span_mse: float = 1000.0 - loom_speed_span_mse_per_sec: float = 3000.0 - loom_speed_decay_sec: float = 0.3 + figure_span_mse: float = 1500.0 + object_memory_sec: float = 10.0 lower_settle_sec: float = 0.35 first_camera: str = '1C' cameras: tuple = ( - CameraButton('1C', 925, 484, 'left', 'figure', pursue_to='2A'), - CameraButton('2A', 983, 599, 'left', 'loom'), + CameraButton('1C', 925, 484, 'left'), ) diff --git a/src/flynaf/env/brain_view.html b/src/flynaf/env/brain_view.html index 7d2d3b1..bcbb9fc 100644 --- a/src/flynaf/env/brain_view.html +++ b/src/flynaf/env/brain_view.html @@ -725,7 +725,7 @@

FLY CONNECTOME · LIVE

} else if (row.kind === 'door') { if (f['door_' + row.side]) { tctx.fillStyle = 'rgba(255,92,74,0.35)'; tctx.fillRect(x, y + 2, dx, inner); } } else if (row.kind === 'tablet') { - if (f.camera) { tctx.fillStyle = f.cam === '2A' ? 'rgba(236,72,153,0.5)' : 'rgba(168,85,247,0.45)'; tctx.fillRect(x, y + 2, dx, inner); } + if (f.camera) { tctx.fillStyle = 'rgba(168,85,247,0.45)'; tctx.fillRect(x, y + 2, dx, inner); } } else if (row.kind === 'feed') { const level = Math.max(f.figure, f.loom); if (level > 0) { diff --git a/src/flynaf/env/overlay.py b/src/flynaf/env/overlay.py index 317ffe8..8f3a9d1 100644 --- a/src/flynaf/env/overlay.py +++ b/src/flynaf/env/overlay.py @@ -27,13 +27,12 @@ def capture_regions() -> list: right = anchor.point(vision.right_target_x, vision.right_target_y) map_corner = anchor.point(camera.map_left, camera.map_top) map_far_corner = anchor.point(camera.map_right, camera.map_bottom) - tablet = config.TABLET_VISION - feed = anchor.point(tablet.feed_x, tablet.feed_y) + view = config.TABLET_VISION.view return [ centered_rect(left[0], left[1], anchor.size(vision.left_bbox_size)), centered_rect(right[0], right[1], anchor.size(vision.right_bbox_size)), (*map_corner, *map_far_corner), - centered_rect(feed[0], feed[1], anchor.size(tablet.feed_size)), + (*anchor.point(view[0], view[1]), *anchor.point(view[2], view[3])), ] @@ -68,18 +67,12 @@ def game_bounds() -> tuple: ) -_TASKBAR_RESERVE = 48 - - -def beside_game_rect( - screen_w: int, screen_h: int, min_width: int, max_width: int, margin: int, -) -> tuple | None: +def beside_game_rect(screen_w: int, min_width: int, max_width: int, margin: int) -> tuple | None: rect = anchor.game_rect() if rect is None: return None - game_x, game_y, game_w, game_h = rect + game_x, game_y, game_w, height = rect top = max(0, game_y) - height = max(game_h, screen_h - top - _TASKBAR_RESERVE) right_space = screen_w - (game_x + game_w) - 2 * margin if right_space >= min_width: width = min(max_width, right_space) diff --git a/src/flynaf/env/tablet_feed.py b/src/flynaf/env/tablet_feed.py index ddd8821..62d028f 100644 --- a/src/flynaf/env/tablet_feed.py +++ b/src/flynaf/env/tablet_feed.py @@ -13,22 +13,101 @@ _log = logging.getLogger(__name__) -def prepare(frame: np.ndarray, pixels: int) -> np.ndarray: +def feed_size(params=None) -> tuple: + params = params or config.TABLET_VISION + left, top, right, bottom = params.view + return ((right - left) // params.view_scale, (bottom - top) // params.view_scale) + + +def prepare(frame: np.ndarray, size: tuple) -> np.ndarray: gray = frame if frame.ndim == 2 else cv2.cvtColor(frame, cv2.COLOR_BGRA2GRAY) - small = cv2.resize(gray.astype(np.float32), (pixels, pixels), interpolation=cv2.INTER_AREA) + small = cv2.resize(gray.astype(np.float32), size, interpolation=cv2.INTER_AREA) return cv2.GaussianBlur(small, (3, 3), 0) -def bank_distance(frame: np.ndarray, bank: list) -> float: +def view_mask(params=None) -> np.ndarray: + params = params or config.TABLET_VISION + width, height = feed_size(params) + left, top = params.view[:2] + scale = params.view_scale + mask = np.ones((height, width), bool) + for hud_left, hud_top, hud_right, hud_bottom in params.hud: + rows = slice(max(0, (hud_top - top) // scale), max(0, -(-(hud_bottom - top) // scale))) + cols = slice(max(0, (hud_left - left) // scale), max(0, -(-(hud_right - left) // scale))) + mask[rows, cols] = False + return mask + + +def _overlap(frame: np.ndarray, references: np.ndarray, mask: np.ndarray, shift: int) -> tuple: + width = frame.shape[1] + if shift >= 0: + ahead, behind = slice(shift, width), slice(0, width - shift) + else: + ahead, behind = slice(0, width + shift), slice(-shift, width) + return frame[:, ahead], references[..., behind], mask[:, ahead] & mask[:, behind] + + +def _cell_means(squared: np.ndarray, mask: np.ndarray, cell: int) -> tuple: + height, width = (squared.shape[-2] // cell) * cell, (squared.shape[-1] // cell) * cell + rows, columns = height // cell, width // cell + lead = squared.shape[:-2] + sums = np.where(mask, squared, 0.0)[..., :height, :width].reshape(*lead, rows, cell, columns, cell) + seen = mask[:height, :width].astype(np.float32).reshape(rows, cell, columns, cell).sum(axis=(1, 3)) + means = sums.sum(axis=(-3, -1)) / np.maximum(seen, 1.0) + return means.reshape(*lead, -1), (seen > 0.6 * cell * cell).ravel() + + +def _global_mse(frame, references, mask, shift) -> np.ndarray: + seen, recorded, shared = _overlap(frame, references, mask, shift) + difference = recorded - seen + return (difference * difference * shared).sum(axis=(-2, -1)) / max(1, int(shared.sum())) + + +def _nearest_alignments(frame, references, mask, params) -> list: + reach = max(0, min(params.pan_reach // params.view_scale, frame.shape[1] // 2)) + coarse = max(1, params.pan_coarse_step) + shifts = list(range(-reach, reach + 1, coarse)) + errors = np.stack([_global_mse(frame, references, mask, s) for s in shifts]) + best = errors.min(axis=0) + chosen = np.argsort(best)[:max(1, params.nearest_references)] + alignments = [] + for index in chosen: + start = shifts[int(errors[:, index].argmin())] + around = range(max(-reach, start - coarse + 1), min(reach, start + coarse - 1) + 1) + reference = references[index] + alignments.append((reference, min(around, key=lambda s: float(_global_mse(frame, reference, mask, s))))) + return alignments + + +def strongest_cells(difference: np.ndarray, mask: np.ndarray, cell: int, count: int) -> float: + means, covered = _cell_means(difference, mask, cell) + means = np.where(covered, means, 0.0) + if means.size == 0: + return float(difference[mask].mean()) if mask.any() else 0.0 + return float(np.sort(means)[::-1][:count].mean()) + + +def bank_distance(frame: np.ndarray, bank: list, mask: np.ndarray | None = None, + params=None) -> float: if not bank: return 0.0 - return min(float(np.mean((frame - reference) ** 2)) for reference in bank) - - -def bank_noise(bank: list) -> float: + params = params or config.TABLET_VISION + mask = np.ones(frame.shape, bool) if mask is None else mask + distances = [] + for reference, shift in _nearest_alignments(frame, np.stack(bank), mask, params): + seen, recorded, shared = _overlap(frame, reference, mask, shift) + distances.append(strongest_cells( + (recorded - seen) ** 2, shared, params.cell_pixels, params.strongest_cells, + )) + return min(distances) + + +def bank_noise(bank: list, mask: np.ndarray | None = None, params=None) -> float: if len(bank) < 2: return 0.0 - return max(bank_distance(frame, bank[:i] + bank[i + 1:]) for i, frame in enumerate(bank)) + return max( + bank_distance(frame, bank[:i] + bank[i + 1:], mask, params) for i, frame in enumerate(bank) + ) def spread_sample(frames: list, count: int) -> list: @@ -41,11 +120,15 @@ def spread_sample(frames: list, count: int) -> list: class TabletFeed: def __init__(self, params=None): self._p = params or config.TABLET_VISION + self._size = feed_size(self._p) + self._mask = view_mask(self._p) self._latest: collections.deque = collections.deque(maxlen=1) self._lock = threading.Lock() self._active = threading.Event() self._banks: dict = {} self._noise: dict = {} + self._watching = None + self._reading = (None, None) threading.Thread(target=self._capture_loop, daemon=True).start() def activate(self): @@ -55,6 +138,8 @@ def deactivate(self): self._active.clear() with self._lock: self._latest.clear() + self._watching = None + self._reading = (None, None) def has_reference(self, camera: str) -> bool: return bool(self._banks.get(camera)) @@ -63,12 +148,10 @@ def noise(self, camera: str) -> float: return self._noise.get(camera, 0.0) def contrast(self, camera: str) -> float | None: - bank = self._banks.get(camera) with self._lock: - frame = self._latest[-1] if self._latest else None - if not bank or frame is None: - return None - return bank_distance(frame, bank) + self._watching = camera + read_camera, read_contrast = self._reading + return read_contrast if read_camera == camera else None def capture_reference(self, camera: str) -> bool: frames = self._collect(self._p.reference_sweep_sec) @@ -77,8 +160,8 @@ def capture_reference(self, camera: str) -> bool: _log.warning('no frames arrived from the tablet while recording camera %s', camera) return False self._banks[camera] = bank - self._noise[camera] = bank_noise(bank) * self._p.noise_margin - _log.info('camera %s reference: %d frames, noise floor %.0f mse', + self._noise[camera] = bank_noise(bank, self._mask, self._p) * self._p.noise_margin + _log.info('camera %s reference: %d frames, noise floor %.0f', camera, len(bank), self._noise[camera]) return True @@ -101,11 +184,22 @@ def _capture_loop(self): with self._lock: if self._active.is_set(): self._latest.append(frame) + camera = self._watching + self._read(camera, frame) time.sleep(self._p.capture_delay_sec) + def _read(self, camera: str | None, frame: np.ndarray): + bank = self._banks.get(camera) + if not bank: + return + contrast = bank_distance(frame, bank, self._mask, self._p) + with self._lock: + if self._watching == camera: + self._reading = (camera, contrast) + def _grab(self, screen) -> np.ndarray: - centre_x, centre_y = anchor.point(self._p.feed_x, self._p.feed_y) - size = anchor.size(self._p.feed_size) - box = {'top': max(0, centre_y - size // 2), 'left': max(0, centre_x - size // 2), - 'width': size, 'height': size} - return prepare(np.array(screen.grab(box)), self._p.feed_pixels) + left, top, right, bottom = self._p.view + x0, y0 = anchor.point(left, top) + x1, y1 = anchor.point(right, bottom) + box = {'top': y0, 'left': x0, 'width': max(1, x1 - x0), 'height': max(1, y1 - y0)} + return prepare(np.array(screen.grab(box)), self._size) diff --git a/src/flynaf/main.py b/src/flynaf/main.py index f1180ff..950c656 100644 --- a/src/flynaf/main.py +++ b/src/flynaf/main.py @@ -15,7 +15,7 @@ from flynaf.night.panel import brain_view_task from flynaf.night.state import SaccadeRequest, SensoryState from flynaf.night.tablet.senses import TabletSenses -from flynaf.night.tablet.watch import TabletWatch, build_gaze +from flynaf.night.tablet.watch import TabletWatch, build_gaze, build_memory from flynaf.night.tasks import engine_task, saccade_task, vision_task from flynaf.recorder import SessionRecorder from flynaf.telemetry import ConnectomeTelemetry @@ -39,11 +39,12 @@ async def _run(telemetry: ConnectomeTelemetry, trace: SessionRecorder, begin_nig feed = SpikeFeed(engine.num_neurons, tracer) saccade = SaccadeRequest() tablet_feed = TabletFeed() - tablet = TabletWatch(build_gaze(), tablet_feed, controller, telemetry, state) + memory = build_memory() + tablet = TabletWatch(build_gaze(), tablet_feed, controller, telemetry, state, memory) async with asyncio.TaskGroup() as tg: tg.create_task(vision_task( - vision, controller, state, shutdown, TabletSenses(tablet_feed), + vision, controller, state, shutdown, TabletSenses(tablet_feed, memory), )) tg.create_task(saccade_task( engine, vision, controller, state, shutdown, telemetry, calibration_done, diff --git a/src/flynaf/night/panel.py b/src/flynaf/night/panel.py index 6dcded6..f6c2184 100644 --- a/src/flynaf/night/panel.py +++ b/src/flynaf/night/panel.py @@ -52,10 +52,10 @@ def restore_game_focus(game: int, panel: int, timeout_sec: float): ) -def _beside_the_game(screen_w: int, screen_h: int): +def _beside_the_game(screen_w: int): params = config.BRAIN_VIEW beside = beside_game_rect( - screen_w, screen_h, params.beside_min_width, params.width, params.ingame_margin, + screen_w, params.beside_min_width, params.width, params.ingame_margin, ) if beside is not None or not params.dock_game: return beside @@ -63,14 +63,14 @@ def _beside_the_game(screen_w: int, screen_h: int): return None _log.info('game window moved to the left edge to make room for the panel') return beside_game_rect( - screen_w, screen_h, params.beside_min_width, params.width, params.ingame_margin, + screen_w, params.beside_min_width, params.width, params.ingame_margin, ) def _choose_panel_placement(url: str): params = config.BRAIN_VIEW screen_w, screen_h = screen_size() - beside = _beside_the_game(screen_w, screen_h) + beside = _beside_the_game(screen_w) if beside is not None: _log.info('brain view beside the game, panel %dx%d at %d,%d', beside[2], beside[3], *beside[:2]) return (*beside, f'{url}?corner={params.corner_radius}', True) diff --git a/src/flynaf/night/tablet/gaze.py b/src/flynaf/night/tablet/gaze.py index fd30c76..bad85e9 100644 --- a/src/flynaf/night/tablet/gaze.py +++ b/src/flynaf/night/tablet/gaze.py @@ -1,13 +1,10 @@ class TabletGaze: - def __init__(self, cameras: tuple, first_camera: str, raise_settle_sec: float, - switch_settle_sec: float): + def __init__(self, cameras: tuple, first_camera: str, raise_settle_sec: float): self._cameras = {camera.name: camera for camera in cameras} self._first = first_camera self._raise_settle = raise_settle_sec - self._switch_settle = switch_settle_sec self.camera = None self.readable_at = 0.0 - self._pursued = False def spec(self, name: str | None = None): return self._cameras.get(name or self.camera) @@ -15,23 +12,15 @@ def spec(self, name: str | None = None): def on_raised(self, now: float) -> str: self.camera = self._first self.readable_at = now + self._raise_settle - self._pursued = False return self.camera def on_lowered(self): self.camera = None - self._pursued = False def readable(self, now: float) -> bool: return self.camera is not None and now >= self.readable_at - def pursue(self, now: float, explore_fired: bool) -> str | None: - current = self.spec() - if current is None or current.pursue_to is None or self._pursued: - return None + def pursued(self, now: float, explore_fired: bool): if not explore_fired or not self.readable(now): return None - self._pursued = True - self.camera = current.pursue_to - self.readable_at = now + self._switch_settle - return self.camera + return self.spec() diff --git a/src/flynaf/night/tablet/memory.py b/src/flynaf/night/tablet/memory.py new file mode 100644 index 0000000..f061a2a --- /dev/null +++ b/src/flynaf/night/tablet/memory.py @@ -0,0 +1,31 @@ +import math + +from flynaf import config + + +class ObjectMemory: + def __init__(self, decay_sec: float, base_rate_hz: float | None = None): + self._decay_sec = decay_sec + rate = base_rate_hz or config.SIMULATION_PARAMS.base_sensory_rate_hz + self._faintest = 1.0 / rate + self._side = None + self._strength = 0.0 + self._at = 0.0 + + def remember(self, side: str, strength: float, now: float) -> bool: + if strength <= self.level(now, side): + return False + self._side, self._strength, self._at = side, strength, now + return True + + def level(self, now: float, side: str | None = None) -> float: + if self._side is None or (side is not None and side != self._side): + return 0.0 + level = self._strength * math.exp(-max(0.0, now - self._at) / self._decay_sec) + return level if level >= self._faintest else 0.0 + + def drive(self, now: float) -> dict: + level = self.level(now) + if level == 0.0: + return {} + return {f'loom_size_{self._side}': level, f'loom_speed_{self._side}': level} diff --git a/src/flynaf/night/tablet/references.py b/src/flynaf/night/tablet/references.py index 76a2067..6b023a7 100644 --- a/src/flynaf/night/tablet/references.py +++ b/src/flynaf/night/tablet/references.py @@ -1,8 +1,8 @@ import asyncio import logging -from flynaf import config -from flynaf.night.monitor import put_tablet, put_tablet_down +from flynaf import clock, config +from flynaf.night.monitor import put_tablet, put_tablet_down, screen_shows from flynaf.night.motor import await_motor _log = logging.getLogger(__name__) @@ -10,7 +10,7 @@ async def capture_camera_references(feed, controller, vision, params=None) -> bool: params = params or config.TABLET_VISION - if not await put_tablet(vision, controller, True, config.CAMERA_DETECTION.flip_confirm_sec): + if not await _raise_at_night_start(vision, controller, params): _log.warning('the camera map did not appear for the camera references, so the fly ' 'will raise the tablet tonight without reading the feed') await _lower(vision, controller) @@ -25,6 +25,17 @@ async def capture_camera_references(feed, controller, vision, params=None) -> bo return all(recorded) +async def _raise_at_night_start(vision, controller, params) -> bool: + started = clock.now() + if await put_tablet(vision, controller, True, config.CAMERA_DETECTION.flip_confirm_sec): + return True + if not await screen_shows(vision, True, params.start_raise_patience_sec): + return False + _log.info('the camera map came up %.1fs after the gesture, the game was still opening the night', + clock.now() - started) + return True + + async def _record(feed, controller, camera: str, params) -> bool: await await_motor(controller.select_camera(camera)) await asyncio.sleep(params.switch_settle_sec) diff --git a/src/flynaf/night/tablet/senses.py b/src/flynaf/night/tablet/senses.py index fc3f912..e353adf 100644 --- a/src/flynaf/night/tablet/senses.py +++ b/src/flynaf/night/tablet/senses.py @@ -1,37 +1,29 @@ from flynaf import config from flynaf.night.state import SensoryState -from flynaf.night.tablet.transduction import LoomChannel, camera_drive, figure_level +from flynaf.night.tablet.transduction import figure_level class TabletSenses: - def __init__(self, feed, params=None): + def __init__(self, feed, memory=None, params=None): self._p = params or config.TABLET_VISION self._feed = feed + self._memory = memory self._cameras = {camera.name: camera for camera in self._p.cameras} - self._loom = LoomChannel( - self._p.loom_span_mse, self._p.loom_speed_span_mse_per_sec, self._p.loom_speed_decay_sec, - ) - self._watching = None def read(self, state: SensoryState, now: float) -> dict: + remembered = self._memory.drive(now) if self._memory is not None else {} + return {**remembered, **self._seen(state, now)} + + def _seen(self, state: SensoryState, now: float) -> dict: camera = state.tablet_camera if state.camera_open else None - if camera != self._watching: - self._watching = camera - self._loom.reset() spec = self._cameras.get(camera) if spec is None or now < state.tablet_readable_at: return {} contrast = self._feed.contrast(camera) if contrast is None: return {} - return camera_drive(spec.channel, spec.side, self._levels(spec, contrast, now)) - - def _levels(self, spec, contrast: float, now: float) -> dict: - noise = self._feed.noise(spec.name) - if spec.channel == 'figure': - return {'figure': figure_level(contrast, noise, self._p.figure_span_mse)} - size, speed = self._loom.update(contrast, noise, now) - return {'size': size, 'speed': speed} + level = figure_level(contrast, self._feed.noise(camera), self._p.figure_span_mse) + return {f'figure_{spec.side}': level} class BlindSenses: diff --git a/src/flynaf/night/tablet/transduction.py b/src/flynaf/night/tablet/transduction.py index 4c973ac..62f5d39 100644 --- a/src/flynaf/night/tablet/transduction.py +++ b/src/flynaf/night/tablet/transduction.py @@ -1,44 +1,6 @@ -import math - - def clamp_unit(value: float) -> float: return min(1.0, max(0.0, value)) def figure_level(contrast: float, noise: float, span: float) -> float: return clamp_unit((contrast - noise) / span) - - -class LoomChannel: - def __init__(self, size_span: float, speed_span_per_sec: float, speed_decay_sec: float): - self._size_span = size_span - self._speed_span = speed_span_per_sec - self._decay_sec = speed_decay_sec - self._previous = None - self._previous_at = 0.0 - self.speed = 0.0 - - def reset(self): - self._previous = None - self.speed = 0.0 - - def update(self, contrast: float, noise: float, now: float) -> tuple: - size = clamp_unit((contrast - noise) / self._size_span) - if self._previous is None: - self._previous, self._previous_at = contrast, now - return size, 0.0 - elapsed = max(now - self._previous_at, 1e-3) - growth = max(0.0, contrast - self._previous) / elapsed - held = self.speed * math.exp(-elapsed / self._decay_sec) - self.speed = max(held, clamp_unit(growth / self._speed_span)) - self._previous, self._previous_at = contrast, now - return size, self.speed - - -def camera_drive(channel: str, side: str, levels: dict) -> dict: - if channel == 'figure': - return {f'figure_{side}': levels.get('figure', 0.0)} - return { - f'loom_size_{side}': levels.get('size', 0.0), - f'loom_speed_{side}': levels.get('speed', 0.0), - } diff --git a/src/flynaf/night/tablet/watch.py b/src/flynaf/night/tablet/watch.py index 24eb5f8..b5ccfca 100644 --- a/src/flynaf/night/tablet/watch.py +++ b/src/flynaf/night/tablet/watch.py @@ -3,21 +3,26 @@ from flynaf import config from flynaf.night.state import SensoryState from flynaf.night.tablet.gaze import TabletGaze +from flynaf.night.tablet.memory import ObjectMemory _log = logging.getLogger(__name__) def build_gaze(params=None) -> TabletGaze: params = params or config.TABLET_VISION - return TabletGaze( - params.cameras, params.first_camera, params.raise_settle_sec, params.switch_settle_sec, - ) + return TabletGaze(params.cameras, params.first_camera, params.raise_settle_sec) + + +def build_memory(params=None) -> ObjectMemory: + params = params or config.TABLET_VISION + return ObjectMemory(params.object_memory_sec) class TabletWatch: def __init__(self, gaze: TabletGaze, feed, controller, telemetry, state: SensoryState, - map_ready_sec: float | None = None): + memory: ObjectMemory | None = None, map_ready_sec: float | None = None): self._gaze = gaze + self._memory = memory or build_memory() self._map_ready_sec = ( config.TABLET_VISION.map_ready_sec if map_ready_sec is None else map_ready_sec ) @@ -32,7 +37,7 @@ def camera(self) -> str | None: def on_raise(self, now: float): self._feed.activate() - self._show(self._gaze.on_raised(now), 'raise', self._map_ready_sec) + self._show(self._gaze.on_raised(now), self._map_ready_sec) def on_lower(self): self._gaze.on_lowered() @@ -43,20 +48,27 @@ def on_lower(self): def update(self, now: float, explore_fired: bool, escape_fired: dict) -> str | None: if self._gaze.camera is None: return None - switched = self._gaze.pursue(now, explore_fired) - if switched is not None: - _log.info('DNp09 fired while watching, gaze follows to camera %s', switched) - self._show(switched, 'pursuit', 0.0) + pursued = self._gaze.pursued(now, explore_fired) + if pursued is not None: + self._remember(pursued, now) for side in ('left', 'right'): if escape_fired.get(side): return side return None - def _show(self, camera: str, cause: str, settle_sec: float): + def _remember(self, spec, now: float): + strength = self._state.tablet_drive.get(f'figure_{spec.side}', 0.0) + fresh = self._memory.level(now, spec.side) == 0.0 + if self._memory.remember(spec.side, strength, now) and fresh: + _log.info('DNp09 fired on camera %s, the figure is remembered on the %s at %.2f', + spec.name, spec.side, strength) + self._telemetry.record_figure_remembered(spec.name, spec.side, strength) + + def _show(self, camera: str, settle_sec: float): self._controller.select_camera(camera, settle_sec) self._state.tablet_camera = camera self._state.tablet_readable_at = self._gaze.readable_at - self._telemetry.record_camera_view(camera, cause) + self._telemetry.record_camera_view(camera) class IdleTablet: diff --git a/src/flynaf/scripts/measure_tablet_pathways.py b/src/flynaf/scripts/measure_tablet_pathways.py index c6b38de..cac7075 100644 --- a/src/flynaf/scripts/measure_tablet_pathways.py +++ b/src/flynaf/scripts/measure_tablet_pathways.py @@ -19,6 +19,9 @@ def conditions() -> list: for level in LEVELS: rows.append((f'loom {level:.2f}, tablet up', {'loom_size_left': level, 'loom_speed_left': level}, rate, 0.0)) + for level in LEVELS: + rows.append((f'loom {level:.2f}, tablet down', + {'loom_size_left': level, 'loom_speed_left': level}, 0.0, 0.0)) for level in LEVELS: rows.append((f'figure {level:.2f}, tablet up', {'figure_left': level}, rate, 0.0)) return rows diff --git a/src/flynaf/telemetry.py b/src/flynaf/telemetry.py index e37c446..0e1a79b 100644 --- a/src/flynaf/telemetry.py +++ b/src/flynaf/telemetry.py @@ -27,7 +27,7 @@ def __init__(self): self.camera_watches = [] self.camera_release_reasons = {'drive': 0, 'search': 0, 'cap': 0, 'escape': 0} self.camera_views = {} - self.camera_view_causes = {'raise': 0, 'pursuit': 0} + self.figures_remembered = {} self.tablet_escapes = {} self.look_contrast = {'left': [], 'right': []} self.look_frames = [] @@ -48,20 +48,24 @@ def record_door_panic(self, side: str, cause: str = 'giant fiber'): self.stats[f'{side}_door_panics'] += 1 self.events.append(f"[{t:>6.1f}s] Visual threat detected. {cause.capitalize()} fired. {side.capitalize()} door slammed.") - def record_camera_view(self, camera: str, cause: str): + def record_camera_view(self, camera: str): t = clock.now() - self.start_time self.camera_views[camera] = self.camera_views.get(camera, 0) + 1 - if cause in self.camera_view_causes: - self.camera_view_causes[cause] += 1 - why = 'the tablet came up on it' if cause == 'raise' else 'DNp09 fired and the gaze followed' - self.events.append(f"[{t:>6.1f}s] Watching CAM {camera}, {why}.") + self.events.append(f"[{t:>6.1f}s] Watching CAM {camera}, the tablet came up on it.") + + def record_figure_remembered(self, camera: str, side: str, strength: float): + t = clock.now() - self.start_time + self.figures_remembered[camera] = self.figures_remembered.get(camera, 0) + 1 + self.events.append( + f"[{t:>6.1f}s] DNp09 fired on a figure in CAM {camera}, remembered on the {side} at {strength:.2f}." + ) def record_tablet_escape(self, side: str, camera: str): t = clock.now() - self.start_time key = f'{camera} {side}' self.tablet_escapes[key] = self.tablet_escapes.get(key, 0) + 1 self.events.append( - f"[{t:>6.1f}s] DNp04 {side} fired on a looming shape in CAM {camera}. Tablet dropped for the door." + f"[{t:>6.1f}s] DNp04 {side} fired while watching CAM {camera}. Tablet dropped for the door." ) def record_door_hold_renewed(self, side: str, cause: str): @@ -171,8 +175,10 @@ def _write_tablet(self, f): return views = ', '.join(f'{camera} x{count}' for camera, count in sorted(self.camera_views.items())) f.write(f"{'Cameras watched':<38}: {views}\n") - causes = self.camera_view_causes - f.write(f"{'Chosen by':<38}: {causes['raise']} on raising, {causes['pursuit']} DNp09 pursuit\n") + remembered = ', '.join( + f'{camera} x{count}' for camera, count in sorted(self.figures_remembered.items()) + ) or 'none' + f.write(f"{'Figures remembered after DNp09':<38}: {remembered}\n") escapes = ', '.join(f'{key} x{count}' for key, count in sorted(self.tablet_escapes.items())) or 'none' f.write(f"{'DNp04 escapes from the tablet':<38}: {escapes}\n\n") diff --git a/tests/test_camera_references.py b/tests/test_camera_references.py new file mode 100644 index 0000000..fb589a9 --- /dev/null +++ b/tests/test_camera_references.py @@ -0,0 +1,92 @@ +import asyncio +import dataclasses +import threading +from unittest.mock import patch + +import pytest + +from flynaf import clock, config +from flynaf.night.tablet import references + + +class LateTablet: + def __init__(self, latency_sec: float | None): + self._latency = latency_sec + self._up_at = None + + def flip(self): + if self.is_camera_up(): + self._up_at = None + elif self._latency is not None: + self._up_at = clock.now() + self._latency + + def is_camera_up(self) -> bool: + return self._up_at is not None and clock.now() >= self._up_at + + def is_camera_down(self) -> bool: + return not self.is_camera_up() + + +class CountingController: + def __init__(self, tablet: LateTablet): + self._tablet = tablet + self.flips = 0 + self.cameras = [] + + def _done(self) -> threading.Event: + done = threading.Event() + done.set() + return done + + def flip_tablet(self) -> threading.Event: + self.flips += 1 + self._tablet.flip() + return self._done() + + def select_camera(self, camera: str, settle_sec: float = 0.0) -> threading.Event: + self.cameras.append(camera) + return self._done() + + +class RecordingFeed: + def activate(self): + pass + + def deactivate(self): + pass + + def capture_reference(self, camera: str) -> bool: + return True + + +@pytest.fixture(autouse=True) +def quick_confirmations(): + detection = dataclasses.replace(config.CAMERA_DETECTION, flip_confirm_sec=0.1, lower_gesture_attempts=1) + with patch.object(config, 'CAMERA_DETECTION', detection): + yield + + +def _params(patience_sec: float): + return dataclasses.replace( + config.TABLET_VISION, start_raise_patience_sec=patience_sec, switch_settle_sec=0.0, + ) + + +@pytest.mark.parametrize('latency_sec, patience_sec, recorded', [ + (0.0, 0.3, True), + (0.25, 0.5, True), + (None, 0.2, False), +]) +def test_a_tablet_that_comes_up_late_at_night_start_is_waited_for_and_never_flipped_twice( + latency_sec, patience_sec, recorded, +): + tablet = LateTablet(latency_sec) + controller = CountingController(tablet) + result = asyncio.run(references.capture_camera_references( + RecordingFeed(), controller, tablet, _params(patience_sec), + )) + assert result is recorded + assert controller.cameras == (['1C'] if recorded else []) + raise_gestures = 1 + lower_gestures = 1 if recorded else 0 + assert controller.flips == raise_gestures + lower_gestures diff --git a/tests/test_inhibition.py b/tests/test_inhibition.py index c59140a..9e85729 100644 --- a/tests/test_inhibition.py +++ b/tests/test_inhibition.py @@ -63,6 +63,12 @@ def test_each_tablet_channel_wakes_its_own_descending_neuron(): assert not crossed, f'{label}: {crossed} answered a channel that is not theirs, {counts}' +def test_a_remembered_figure_keeps_the_giant_fiber_answering_once_the_tablet_is_down(): + faint = {'loom_size_left': 0.05, 'loom_speed_left': 0.05} + counts = _spike_frames(SensoryState(tablet_drive=faint), TABLET_FRAMES) + assert counts['giant_fiber'] > 0, f'the giant fiber ignored the remembered figure, {counts}' + + if __name__ == '__main__': for name, fn in sorted(globals().items()): if name.startswith('test_'): diff --git a/tests/test_monitor_lowering.py b/tests/test_monitor_lowering.py index 18d0229..59701af 100644 --- a/tests/test_monitor_lowering.py +++ b/tests/test_monitor_lowering.py @@ -198,7 +198,7 @@ async def scenario(): assert bool(closed) is door_closed -@pytest.mark.parametrize('watched, still_open', [(1.0, True), (1.9, True), (2.1, False)]) +@pytest.mark.parametrize('watched, still_open', [(1.0, True), (3.1, True), (3.3, False)]) def test_the_fly_only_tires_of_a_camera_after_it_could_read_it(watched, still_open): screen = Screen() control_, state, _, _ = control(screen) diff --git a/tests/test_object_memory.py b/tests/test_object_memory.py new file mode 100644 index 0000000..25cafb6 --- /dev/null +++ b/tests/test_object_memory.py @@ -0,0 +1,50 @@ +import math + +from flynaf.night.tablet.memory import ObjectMemory + +DECAY = 4.0 +BASE_RATE = 1000.0 + + +def _memory() -> ObjectMemory: + return ObjectMemory(DECAY, BASE_RATE) + + +def test_a_remembered_figure_fades_with_the_memory_time_constant(): + memory = _memory() + memory.remember('left', 0.6, now=10.0) + assert memory.level(10.0) == 0.6 + assert abs(memory.level(14.0) - 0.6 * math.exp(-1.0)) < 1e-9 + assert memory.level(14.0, 'right') == 0.0 + + +def test_the_memory_is_gone_once_it_would_feed_less_than_one_spike_a_second(): + memory = _memory() + memory.remember('left', 0.6, now=0.0) + vanishes_at = DECAY * math.log(0.6 * BASE_RATE) + assert memory.level(vanishes_at - 0.1) > 0.0 + assert memory.level(vanishes_at + 0.1) == 0.0 + assert memory.drive(vanishes_at + 0.1) == {} + + +def test_a_stronger_sighting_refreshes_the_memory_and_a_weaker_one_does_not(): + memory = _memory() + assert memory.remember('left', 0.4, now=0.0) + assert not memory.remember('left', 0.3, now=0.5) + assert memory.remember('left', 0.7, now=1.0) + assert memory.level(1.0) == 0.7 + assert not memory.remember('left', 0.0, now=2.0) + + +def test_a_faded_memory_is_refreshed_by_any_new_sighting(): + memory = _memory() + memory.remember('left', 0.6, now=0.0) + assert memory.remember('left', 0.2, now=16.0) + assert memory.level(16.0) == 0.2 + + +def test_the_memory_drives_the_looming_populations_on_its_side(): + memory = _memory() + assert memory.drive(0.0) == {} + memory.remember('left', 0.5, now=0.0) + assert memory.drive(0.0) == {'loom_size_left': 0.5, 'loom_speed_left': 0.5} diff --git a/tests/test_overlay_placement.py b/tests/test_overlay_placement.py index cb5adbf..f87b119 100644 --- a/tests/test_overlay_placement.py +++ b/tests/test_overlay_placement.py @@ -3,7 +3,7 @@ import pytest from flynaf import config -from flynaf.env import anchor +from flynaf.env import anchor, overlay from flynaf.env.overlay import ( beside_game_rect, capture_regions, @@ -25,17 +25,29 @@ def unanchored(): anchor.set_game_rect(None) +@pytest.fixture +def tablet_view_ignored(): + regions = [region for region in capture_regions() if region != _tablet_view()] + with patch.object(overlay, 'capture_regions', return_value=regions): + yield + + +def _tablet_view() -> tuple: + left, top, right, bottom = config.TABLET_VISION.view + return (*anchor.point(left, top), *anchor.point(right, bottom)) + + def _beside(game_rect, screen=(1920, 1080)): with patch.object(anchor, 'game_rect', return_value=game_rect): - return beside_game_rect(screen[0], screen[1], 420, 620, 8) + return beside_game_rect(screen[0], 420, 620, 8) -def test_the_panel_sits_right_of_a_windowed_game_down_to_the_taskbar(): - assert _beside((0, 31, 1280, 720)) == (1288, 31, 620, 1001) +def test_the_panel_sits_right_of_a_windowed_game_at_the_game_height(): + assert _beside((0, 31, 1280, 720)) == (1288, 31, 620, 720) def test_the_panel_takes_the_left_side_when_only_that_one_fits(): - assert _beside((600, 31, 1280, 720)) == (8, 31, 584, 1001) + assert _beside((600, 31, 1280, 720)) == (8, 31, 584, 720) def test_no_panel_beside_a_game_that_fills_the_screen(): @@ -101,14 +113,19 @@ def test_real_config_regions_leave_room_on_this_layout(): ) -def test_ingame_overlay_clears_every_capture_and_motor_region(): +def test_no_ingame_overlay_over_the_camera_view_the_tablet_reads(): + params = config.BRAIN_VIEW + assert ingame_overlay_rect(params.ingame_width, params.ingame_height, params.ingame_margin) is None + + +def test_ingame_overlay_clears_every_capture_and_motor_region(tablet_view_ignored): params = config.BRAIN_VIEW position = ingame_overlay_rect(params.ingame_width, params.ingame_height, params.ingame_margin) assert position is not None x, y = position panel = (x, y, x + params.ingame_width, y + params.ingame_height) - for region in capture_regions() + motor_regions(): + for region in overlay.capture_regions() + motor_regions(): assert not rects_overlap(panel, region) @@ -118,7 +135,7 @@ def test_ingame_panel_is_wide_enough_for_the_browser_to_honour_it(): assert config.BRAIN_VIEW.ingame_width >= EDGE_MIN_WIDTH -def test_ingame_overlay_stays_inside_the_game_area(): +def test_ingame_overlay_stays_inside_the_game_area(tablet_view_ignored): params = config.BRAIN_VIEW left, _top, right, bottom = game_bounds() x, y = ingame_overlay_rect(params.ingame_width, params.ingame_height, params.ingame_margin) @@ -133,7 +150,7 @@ def test_ingame_overlay_returns_none_when_the_panel_cannot_fit(): assert ingame_overlay_rect(right - left + 1, 100, margin=1) is None -def test_ingame_overlay_prefers_the_top_of_the_screen(): +def test_ingame_overlay_prefers_the_top_of_the_screen(tablet_view_ignored): params = config.BRAIN_VIEW _x, y = ingame_overlay_rect(params.ingame_width, params.ingame_height, params.ingame_margin) diff --git a/tests/test_tablet_escape.py b/tests/test_tablet_escape.py index 1e54026..acbf367 100644 --- a/tests/test_tablet_escape.py +++ b/tests/test_tablet_escape.py @@ -50,7 +50,7 @@ def quick_confirmation(monkeypatch): class StubTablet: - camera = '2A' + camera = '1C' def __init__(self, escape_side): self.escape_side = escape_side @@ -117,7 +117,7 @@ def test_the_escape_is_reported_as_its_own_release_and_door_cause(): control, doors, controller, telemetry, state, tablet = _run('right') assert telemetry.camera_release_reasons['escape'] == 1 assert telemetry.stats['right_door_panics'] == 1 - assert telemetry.tablet_escapes == {'2A right': 1} + assert telemetry.tablet_escapes == {'1C right': 1} assert tablet.lowered >= 1 diff --git a/tests/test_tablet_gaze.py b/tests/test_tablet_gaze.py index e0935f5..c101c9e 100644 --- a/tests/test_tablet_gaze.py +++ b/tests/test_tablet_gaze.py @@ -1,15 +1,15 @@ from flynaf import config from flynaf.night.state import SensoryState from flynaf.night.tablet.gaze import TabletGaze +from flynaf.night.tablet.memory import ObjectMemory from flynaf.night.tablet.watch import TabletWatch from flynaf.telemetry import ConnectomeTelemetry RAISE = 1.2 -SWITCH = 0.5 def _gaze() -> TabletGaze: - return TabletGaze(config.TABLET_VISION.cameras, '1C', RAISE, SWITCH) + return TabletGaze(config.TABLET_VISION.cameras, '1C', RAISE) class _Done: @@ -48,7 +48,7 @@ def test_raising_the_tablet_always_opens_on_the_cove(): PURSUIT_CASES = ( ('no spike', 20.0, False, None), ('spike during the raise static', 10.1, True, None), - ('spike once the cove is readable', 20.0, True, '2A'), + ('spike once the cove is readable', 20.0, True, '1C'), ) @@ -56,19 +56,40 @@ def test_pursuit_needs_a_dnp09_spike_on_a_readable_cove(): for label, now, fired, expected in PURSUIT_CASES: gaze = _gaze() gaze.on_raised(now=10.0) - assert gaze.pursue(now, fired) == expected, label + pursued = gaze.pursued(now, fired) + assert (pursued.name if pursued else None) == expected, label -def test_pursuit_happens_once_per_raise_and_never_from_the_hall(): +def test_the_gaze_never_leaves_the_cove_on_its_own(): gaze = _gaze() gaze.on_raised(now=0.0) - assert gaze.pursue(5.0, True) == '2A' - assert gaze.camera == '2A' - assert gaze.pursue(10.0, True) is None + gaze.pursued(5.0, True) + assert gaze.camera == '1C' - gaze.on_lowered() - gaze.on_raised(now=20.0) - assert gaze.pursue(25.0, True) == '2A' + +def _watching_the_cove(figure: float): + state = SensoryState(camera_open=True) + controller, telemetry = RecordingController(), ConnectomeTelemetry() + memory = ObjectMemory(decay_sec=4.0) + watch = TabletWatch(_gaze(), RecordingFeed(), controller, telemetry, state, memory) + watch.on_raise(now=0.0) + state.tablet_drive = {'figure_left': figure} + return watch, memory, controller, telemetry + + +def test_a_dnp09_spike_on_the_cove_remembers_the_figure_it_was_looking_at(): + watch, memory, controller, telemetry = _watching_the_cove(figure=0.6) + watch.update(5.0, explore_fired=True, escape_fired={}) + assert abs(memory.level(5.0, 'left') - 0.6) < 1e-9 + assert controller.selected == [('1C', config.TABLET_VISION.map_ready_sec)] + assert telemetry.figures_remembered == {'1C': 1} + + +def test_a_spontaneous_dnp09_spike_on_an_empty_view_remembers_nothing(): + watch, memory, controller, telemetry = _watching_the_cove(figure=0.0) + watch.update(5.0, explore_fired=True, escape_fired={}) + assert memory.level(5.0) == 0.0 + assert telemetry.figures_remembered == {} def test_the_watch_taps_the_map_and_tells_the_senses_where_it_looks(): @@ -80,11 +101,7 @@ def test_the_watch_taps_the_map_and_tells_the_senses_where_it_looks(): assert feed.active assert state.tablet_camera == '1C' assert state.tablet_readable_at == RAISE - - watch.update(5.0, explore_fired=True, escape_fired={}) - assert controller.selected == [('1C', config.TABLET_VISION.map_ready_sec), ('2A', 0.0)] - assert state.tablet_camera == '2A' - assert telemetry.camera_view_causes == {'raise': 1, 'pursuit': 1} + assert telemetry.camera_views == {'1C': 1} watch.on_lower() assert not feed.active diff --git a/tests/test_tablet_senses.py b/tests/test_tablet_senses.py index df40934..6533a8f 100644 --- a/tests/test_tablet_senses.py +++ b/tests/test_tablet_senses.py @@ -1,4 +1,7 @@ +import math + from flynaf.night.state import SensoryState +from flynaf.night.tablet.memory import ObjectMemory from flynaf.night.tablet.senses import TabletSenses @@ -22,7 +25,7 @@ def _watching(camera, readable_at=0.0, camera_open=True) -> SensoryState: ('tablet down', _watching('1C', camera_open=False)), ('no camera chosen', _watching(None)), ('camera still in its switching static', _watching('1C', readable_at=99.0)), - ('a camera with no reference', _watching('2A')), + ('a camera the fly does not read', _watching('2A')), ) @@ -33,30 +36,28 @@ def test_nothing_reaches_the_brain_unless_a_known_camera_is_readable(): def test_the_cove_drives_the_figure_population_on_its_side(): - senses = TabletSenses(ScriptedFeed({'1C': 600.0})) + senses = TabletSenses(ScriptedFeed({'1C': 850.0})) drive = senses.read(_watching('1C'), now=1.0) assert drive == {'figure_left': 0.5} -def test_the_west_hall_drives_the_looming_populations(): - feed = ScriptedFeed({'2A': 100.0}) - senses = TabletSenses(feed) - state = _watching('2A') - senses.read(state, now=1.0) - feed.contrasts['2A'] = 1100.0 - drive = senses.read(state, now=1.1) - assert set(drive) == {'loom_size_left', 'loom_speed_left'} - assert drive['loom_size_left'] == 1.0 - assert drive['loom_speed_left'] > 0.9 - - -def test_switching_cameras_does_not_count_the_cut_as_motion(): - feed = ScriptedFeed({'1C': 100.0, '2A': 5000.0}) - senses = TabletSenses(feed) - senses.read(_watching('2A'), now=0.0) - senses.read(_watching('1C'), now=0.5) - drive = senses.read(_watching('2A'), now=1.0) - assert drive['loom_speed_left'] == 0.0 +def _remembering(strength: float, now: float = 0.0) -> ObjectMemory: + memory = ObjectMemory(decay_sec=4.0) + memory.remember('left', strength, now) + return memory + + +def test_a_remembered_figure_drives_the_looming_populations_on_its_side_with_the_tablet_down(): + senses = TabletSenses(ScriptedFeed({'1C': 5000.0}), _remembering(0.6)) + drive = senses.read(_watching('1C', camera_open=False), now=0.0) + assert drive == {'loom_size_left': 0.6, 'loom_speed_left': 0.6} + + +def test_the_remembered_figure_and_the_cove_reach_the_brain_together(): + senses = TabletSenses(ScriptedFeed({'1C': 850.0}), _remembering(0.6)) + drive = senses.read(_watching('1C'), now=4.0) + assert drive['figure_left'] == 0.5 + assert abs(drive['loom_size_left'] - 0.6 * math.exp(-1.0)) < 1e-9 if __name__ == '__main__': diff --git a/tests/test_tablet_transduction.py b/tests/test_tablet_transduction.py index 259e72f..51b62d7 100644 --- a/tests/test_tablet_transduction.py +++ b/tests/test_tablet_transduction.py @@ -1,7 +1,16 @@ +import cv2 import numpy as np -from flynaf.env.tablet_feed import bank_distance, bank_noise, prepare, spread_sample -from flynaf.night.tablet.transduction import LoomChannel, camera_drive, figure_level +from flynaf import config +from flynaf.env.tablet_feed import ( + bank_distance, + bank_noise, + feed_size, + prepare, + spread_sample, + view_mask, +) +from flynaf.night.tablet.transduction import figure_level FIGURE_CASES = ( (0.0, 100.0, 1000.0, 0.0), @@ -17,82 +26,75 @@ def test_figure_level_is_the_contrast_above_noise_over_the_span(): assert abs(figure_level(contrast, noise, span) - expected) < 1e-9, (contrast, noise) -def _loom() -> LoomChannel: - return LoomChannel(size_span=1000.0, speed_span_per_sec=3000.0, speed_decay_sec=0.3) - - -def test_a_growing_shape_drives_speed_and_a_standing_one_only_size(): - loom = _loom() - loom.update(0.0, 0.0, now=0.0) - size, speed = loom.update(600.0, 0.0, now=0.1) - assert size == 0.6 - assert speed > 0.9 - - for step in range(2, 20): - size, speed = loom.update(600.0, 0.0, now=step * 0.1) - assert size == 0.6 - assert speed < 0.01 +def _scene(seed: int = 3) -> np.ndarray: + rng = np.random.default_rng(seed) + width, height = feed_size() + return cv2.GaussianBlur(rng.uniform(0, 120, (height, width)).astype(np.float32), (5, 5), 0) -def test_speed_is_held_briefly_so_a_slow_engine_frame_can_still_read_it(): - loom = _loom() - loom.update(0.0, 0.0, now=0.0) - _, peak = loom.update(900.0, 0.0, now=0.05) - _, after = loom.update(900.0, 0.0, now=0.10) - assert 0.0 < after < peak +def _panned(scene: np.ndarray, columns: int) -> np.ndarray: + return np.roll(scene, columns, axis=1) -def test_a_shape_that_shrinks_is_not_looming(): - loom = _loom() - loom.update(900.0, 0.0, now=0.0) - _, speed = loom.update(100.0, 0.0, now=0.1) - assert speed == 0.0 - - -def test_reset_forgets_the_previous_camera(): - loom = _loom() - loom.update(0.0, 0.0, now=0.0) - loom.update(900.0, 0.0, now=0.1) - loom.reset() - _, speed = loom.update(900.0, 0.0, now=0.2) - assert speed == 0.0 +def test_bank_distance_takes_the_nearest_reference(): + left = np.zeros((10, 10), np.float32) + right = np.full((10, 10), 50.0, np.float32) + assert bank_distance(right, [left, right]) == 0.0 + assert bank_distance(np.full((10, 10), 45.0, np.float32), [left, right]) == 25.0 + assert bank_distance(left, []) == 0.0 -DRIVE_CASES = ( - ('figure', 'left', {'figure': 0.4}, {'figure_left': 0.4}), - ('loom', 'left', {'size': 0.2, 'speed': 0.7}, {'loom_size_left': 0.2, 'loom_speed_left': 0.7}), - ('loom', 'right', {}, {'loom_size_right': 0.0, 'loom_speed_right': 0.0}), -) +def test_the_camera_pan_is_aligned_away_before_comparing(): + scene = _scene() + bank = [scene[:, 20:60]] + for columns in (-12, -5, 0, 7, 15): + assert bank_distance(_panned(scene, columns)[:, 20:60], bank) < 1.0, columns -def test_camera_drive_names_the_population_on_the_camera_side(): - for channel, side, levels, expected in DRIVE_CASES: - assert camera_drive(channel, side, levels) == expected, (channel, side) +def test_a_small_object_stands_out_instead_of_being_diluted_by_the_frame(): + scene = _scene() + figure = scene.copy() + figure[5:12, 30:37] = 255.0 + whole_frame_mse = float(np.mean((figure - scene) ** 2)) + assert bank_distance(figure, [scene]) > 10 * whole_frame_mse -def test_bank_distance_takes_the_nearest_reference_so_panning_is_not_a_threat(): - left = np.zeros((8, 8), np.float32) - right = np.full((8, 8), 50.0, np.float32) - assert bank_distance(right, [left, right]) == 0.0 - assert bank_distance(np.full((8, 8), 45.0, np.float32), [left, right]) == 25.0 - assert bank_distance(left, []) == 0.0 +def test_changes_under_the_hud_mask_are_ignored(): + scene = _scene() + clock_changed = scene.copy() + clock_changed[:10, 60:] = 255.0 + mask = np.ones(scene.shape, bool) + mask[:10, 60:] = False + assert bank_distance(clock_changed, [scene], mask) < 1.0 + assert bank_distance(clock_changed, [scene]) > 100.0 def test_bank_noise_is_the_worst_leave_one_out_distance(): - frames = [np.full((4, 4), value, np.float32) for value in (0.0, 1.0, 3.0)] + frames = [np.full((10, 10), value, np.float32) for value in (0.0, 1.0, 3.0)] assert bank_noise(frames) == 4.0 assert bank_noise(frames[:1]) == 0.0 +def test_the_view_mask_hides_the_camera_map_and_keeps_the_middle_of_the_feed(): + mask = view_mask() + width, height = feed_size() + assert mask.shape == (height, width) + left, top, right, bottom = config.TABLET_VISION.view + scale = config.TABLET_VISION.view_scale + assert not mask[(400 - top) // scale, (1000 - left) // scale] + assert mask[(200 - top) // scale, (900 - left) // scale] + assert mask[(330 - top) // scale, (470 - left) // scale] + + def test_spread_sample_keeps_the_whole_sweep(): assert spread_sample(list(range(10)), 20) == list(range(10)) assert spread_sample(list(range(100)), 4) == [0, 25, 50, 75] -def test_prepare_shrinks_colour_frames_to_a_square_grey_patch(): - frame = np.random.default_rng(1).integers(0, 255, (120, 120, 4), dtype=np.uint8) - patch = prepare(frame, 32) - assert patch.shape == (32, 32) +def test_prepare_shrinks_colour_frames_to_the_feed_size(): + frame = np.random.default_rng(1).integers(0, 255, (120, 200, 4), dtype=np.uint8) + patch = prepare(frame, (50, 30)) + assert patch.shape == (30, 50) assert patch.dtype == np.float32