Implement the mapping API

README.md

@@ -92,11 +92,19 @@ used for your connection.
     On a different shell check the `rostopic list` output is:
 
     ``` bash
+    /cortex/download_scan
+    /cortex/hovermap_status
     /cortex/lidar/corrected
     /cortex/mule_bridge/status
     /cortex/occupancy_grid_map/configuration
     /cortex/occupancy_grid_map/data
     /cortex/odometry
+    /cortex/scan_download_successful
+    /cortex/scan_names_request
+    /cortex/scan_names_response
+    /cortex/set_scan_prefix
+    /cortex/start_scan
+    /cortex/stop_scan
     /cortex/tf
     /cortex/tf_static
     /rosout
@@ -120,12 +128,20 @@ used for your connection.
 | cortex/odometry                   | nav_msgs/Odometry      | SLAM corrected odometry          | 100               |                  |
 | cortex/tf                         | tf_msgs/TFMessage      | Non-static transform             | Variable          |                  |
 | cortex/tf_static                  | tf_msgs/TFMessage      | Static transforms                | Once (latched)    |                  |
+| cortex/hovermap_status            | hovermap_api_msgs/HovermapStatus | Hovermap and current scan status | 1        |                  |
+| cortex/scan_names_response        | hovermap_api_msgs/ScanInformationList | List of scans stored on the Hovermap newest to oldest | On request | Reply to `cortex/scan_names_request` |
+| cortex/scan_download_successful   | std_msgs/Bool          | Outcome of a scan download: `true` on success, `false` on failure | On download completion | Published after each `cortex/download_scan` request finishes |
 
 ### Sent from client
 
 | Topic name                                | Type                   | Description                        |
 |-------------------------------------------|------------------------|------------------------------------|
 | cortex/occupancy_grid_map/configuration   | std_msgs/String        | Occupancy grid config YAML endpoint|
+| cortex/scan_names_request                 | std_msgs/Empty         | Request the list of stored scans   |
+| cortex/set_scan_prefix                    | std_msgs/String        | Set the prefix used to name new scans |
+| cortex/start_scan                         | std_msgs/Empty         | Start a mapping scan                |
+| cortex/stop_scan                          | std_msgs/Empty         | Stop the current mapping scan       |
+| cortex/download_scan                      | std_msgs/String        | Download a stored scan by name      |
 
 ### Topic details
 
@@ -169,6 +185,34 @@ used for your connection.
     - The configuration is sent to the Hovermap as a string representing a YAML file. A node is provided to do this.
     - The custom configuration is only used when using external_api, and is persistent across runs
 
+6. Hovermap status (`cortex/hovermap_status`)
+    - Reports the current status of the Hovermap and any scan in progress. The `HovermapStatus` message contains:
+        - `scan_prefix`: the prefix currently applied when naming new scans (see `cortex/set_scan_prefix`)
+        - `current_scan_name`: the name of the current or most recent scan
+        - `free_space`: the storage available on the Hovermap, in bytes
+        - `scan_running`: `true` while a scan is in progress (including while it is starting up), `false` otherwise
+
+7. Setting the scan prefix (`cortex/set_scan_prefix`)
+    - Publish a `std_msgs/String` to `cortex/set_scan_prefix` to set the prefix used when naming new scans. New scans are named `<prefix>_NN` where NN is a number starting at 01 and incremented after each scan is started.  The prefix is persistent across boots.
+
+8. Starting and stopping a scan (`cortex/start_scan`, `cortex/stop_scan`)
+    - Publish a `std_msgs/Empty` to `cortex/start_scan` to start a mapping scan, or to `cortex/stop_scan` to stop the current scan.
+    - Starting and stopping can take a few seconds
+
+9. Listing stored scans (`cortex/scan_names_request` → `cortex/scan_names_response`)
+    - Publish a `std_msgs/Empty` to `cortex/scan_names_request` to request the list of scans stored on the Hovermap
+    - The Hovermap replies on `cortex/scan_names_response` with a `ScanInformationList`, ordered newest first.  `ScanInformationList` and its constituent `ScanInformation` messages are custom messages available in `hovermap_api_msgs`. Each `ScanInformation` entry contains:
+        - `name`: the scan name, as used by `cortex/download_scan`
+        - `size`: the size of the scan on the Hovermap, in bytes (the downloaded archive is compressed and will be smaller)
+
+10. Downloading a scan (`cortex/download_scan`)
+    - Publish a `std_msgs/String` containing a scan `name` (from `cortex/scan_names_response`) to download that scan
+    - Scans with duration greater than 1.5 hours may cause the bridge to timeout while waiting for hovermap to prepare the scan for download.  These scans can still be downloaded manually using a USB key or the webui.
+    - The scan is saved as `<name>.zip` in `/data/downloads`. Bind-mount a host directory to this location by passing it to `run_docker` (e.g. `./docker/run_docker ~/hovermap_scans`) to retrieve downloads on the client machine.
+    - A scan cannot be downloaded while a scan is running, or while another download is already in progress
+    - Download progress is reported in rosout
+    - When the download finishes, a `std_msgs/Bool` is published on `cortex/scan_download_successful` reporting the outcome: `true` if the scan was downloaded successfully, `false` if it failed (e.g. a scan is running, another download is in progress, or a network error occurred)
+
 ## Time Synchronisation
 
 The Hovermap is configured to act as an NTP server to allow API users synchronise the client's clock to the Hovermap's by:

docker/Dockerfile

@@ -16,9 +16,12 @@ RUN apt update && apt install -y \
     python3-catkin-tools \
     python3-pip
 
+# Install pip dependencies (only rebuilds if requirements.txt changes)
+RUN --mount=type=bind,src=src/mule_bridge/mule_bridge/requirements.txt,dst=/tmp/requirements.txt \
+    python3 -m pip install -r /tmp/requirements.txt
+
 # Build the repo into the context
 RUN --mount=type=bind,src=src/,dst=/ros_ws/src \
-    python3 -m pip install -r src/mule_bridge/mule_bridge/requirements.txt && \
     source /opt/ros/noetic/setup.bash && \
     catkin init && \
     catkin config --install && \

docker/run_docker

@@ -2,9 +2,38 @@
 
 # Script to run the Docker container for the hovermap_ros_api.
 # ensure using the host network to prevent connection issues.
+# Usage: run_docker [download_directory]
 
 set -euo pipefail
 
+usage() {
+    echo "Usage: $(basename "$0") [download_directory]"
+    echo ""
+    echo "Run the hovermap_ros_api Docker container."
+    echo ""
+    echo "Arguments:"
+    echo "  download_directory  Optional host directory to bind-mount for scan downloads"
+    echo ""
+    echo "Options:"
+    echo "  -h, --help          Show this help message and exit"
+}
+
+if [[ ${1:-} == "-h" || ${1:-} == "--help" ]]; then
+    usage
+    exit 0
+fi
+
 IMAGE_NAME="hovermap_ros_api_image"
 REPO_ROOT="$(dirname "$(realpath "$0")")/.."
-docker run -it --rm --network=host --mount type=bind,src="$REPO_ROOT/src",dst=/ros_ws/src "${IMAGE_NAME}"
+
+EXTRA_MOUNTS=""
+if [[ $# -ge 1 ]]; then
+    DOWNLOAD_DIR="$(realpath "$1")"
+    mkdir -p "$DOWNLOAD_DIR"
+    EXTRA_MOUNTS="--mount type=bind,src=$DOWNLOAD_DIR,dst=/data/downloads"
+fi
+
+docker run -it --rm --network=host \
+    --mount type=bind,src="$REPO_ROOT/src",dst=/ros_ws/src \
+    $EXTRA_MOUNTS \
+    "${IMAGE_NAME}"

src/hovermap_api/CMakeLists.txt

@@ -2,12 +2,14 @@ cmake_minimum_required(VERSION 3.16.3)
 project(hovermap_api)
 
 find_package(catkin REQUIRED COMPONENTS
+  hovermap_api_msgs
   nav_msgs
   sensor_msgs
   tf2_ros
 )
 
 catkin_package(CATKIN_DEPENDS
+  hovermap_api_msgs
   nav_msgs
   sensor_msgs
   tf2_ros
@@ -16,6 +18,7 @@ catkin_package(CATKIN_DEPENDS
 catkin_install_python(
     PROGRAMS
     scripts/configure_perception
+    scripts/http_interface_node
     DESTINATION 
     ${CATKIN_PACKAGE_BIN_DESTINATION})
 

src/hovermap_api/launch/api.launch

@@ -3,4 +3,8 @@
     <node pkg="mule_bridge" name="mule_bridge" type="mule_bridge" required="true" output="screen" ns="cortex">
         <rosparam command="load" file="$(find hovermap_api)/config/mule.yaml"/>
     </node>
+
+    <node pkg="hovermap_api" name="http_interface_node" type="http_interface_node" output="screen" ns="cortex">
+        <rosparam command="load" file="$(find hovermap_api)/config/mule.yaml"/>
+    </node>
 </launch>

src/hovermap_api/package.xml

@@ -11,6 +11,7 @@
   <!-- Dependencies -->
   <buildtool_depend>catkin</buildtool_depend>
   <exec_depend>mule_bridge</exec_depend>
+  <depend>hovermap_api_msgs</depend>
   <depend>nav_msgs</depend>
   <depend>sensor_msgs</depend>
   <depend>tf2_ros</depend>

src/hovermap_api/scripts/http_interface_node

@@ -0,0 +1,187 @@
+#!/usr/bin/python3
+import json
+import os
+import threading
+import rospy
+import urllib.request
+import urllib.error
+from std_msgs.msg import Bool, Empty, String
+from hovermap_api_msgs.msg import ScanInformation, ScanInformationList, HovermapStatus
+
+# Maps ip_prefix from mule.yaml to the actual Hovermap address
+IP_PREFIX_TO_ADDRESS = {
+    "192.168.2.0": "192.168.2.115",
+    "192.168.3.0": "192.168.3.115",
+    "10.9.0.0": "10.9.0.1",
+}
+
+DOWNLOAD_DIR = "/data/downloads"
+
+STATUS_POLL_INTERVAL = 1  # seconds
+SCAN_DOWNLOAD_PREP_TIMEOUT = 450.0  # seconds to wait for zip file to be generated
+
+# Payload states (from the /status "state" field) that do NOT correspond to a
+# running scan. Every other state is a scan actively running or transitioning
+# through the start/stop sequence, so scan_running is True for them.
+NON_RUNNING_STATES = frozenset(
+    {
+        "Stopped",
+        "Disabled",
+        "Booting",
+        "USB Error",
+        "USB Mounted",
+        "Transferring Data",
+        "Processes Stopped",
+    }
+)
+
+
+class HttpInterfaceNode:
+    def __init__(self):
+        rospy.init_node("http_interface_node")
+
+        ip_prefix = rospy.get_param("~ip_prefix")
+        assert isinstance(ip_prefix, str)  # To make pyright happy
+        if ip_prefix not in IP_PREFIX_TO_ADDRESS:
+            rospy.logfatal(f"Unknown ip_prefix: {ip_prefix}")
+            raise RuntimeError(f"Unknown ip_prefix: {ip_prefix}")
+
+        self._base_url = f"http://{IP_PREFIX_TO_ADDRESS[ip_prefix]}"
+
+        os.makedirs(DOWNLOAD_DIR, exist_ok=True)
+
+        rospy.Subscriber("start_scan", Empty, self._on_start_scan)
+        rospy.Subscriber("stop_scan", Empty, self._on_stop_scan)
+        rospy.Subscriber("scan_names_request", Empty, self._on_scan_names_request)
+        rospy.Subscriber("download_scan", String, self._on_download_scan)
+        rospy.Subscriber("set_scan_prefix", String, self._on_set_scan_prefix)
+
+        self._scan_names_pub = rospy.Publisher(
+            "scan_names_response", ScanInformationList, queue_size=5
+        )
+        self._status_pub = rospy.Publisher(
+            "hovermap_status", HovermapStatus, queue_size=5
+        )
+        self._scan_download_successful_pub = rospy.Publisher(
+            "scan_download_successful", Bool, queue_size=5
+        )
+
+        rospy.Timer(rospy.Duration(STATUS_POLL_INTERVAL), self._poll_status)
+
+        rospy.loginfo(f"http_interface_node ready, target: {self._base_url}")
+
+    def _get(self, path, log=True):
+        url = f"{self._base_url}{path}"
+        try:
+            req = urllib.request.Request(url)
+            with urllib.request.urlopen(req, timeout=5.0) as response:
+                body = response.read().decode()
+                if log:
+                    rospy.loginfo(f"GET {url} -> {response.status}: {body}")
+                return body
+        except urllib.error.HTTPError as e:
+            rospy.logerr(f"GET {url} failed: HTTP {e.code} {e.reason}")
+            return None
+        except urllib.error.URLError as e:
+            rospy.logerr(f"GET {url} failed: {e.reason}")
+            return None
+
+    def _on_start_scan(self, msg):
+        self._get("/startsystem?mission_type=1")
+
+    def _on_stop_scan(self, msg):
+        self._get("/stopsystem")
+
+    def _on_scan_names_request(self, msg):
+        body = self._get("/files")
+        if body is None:
+            return
+        try:
+            data = json.loads(body)
+            scans = sorted(data.get("scans", []), key=lambda s: s["time"], reverse=True)
+            scan_list = ScanInformationList()
+            for scan in scans:
+                info = ScanInformation()
+                info.name = scan["scan"]
+                info.size = scan["size"]
+                scan_list.scans.append(info)  # pyright: ignore[reportOptionalMemberAccess]
+            self._scan_names_pub.publish(scan_list)
+        except (json.JSONDecodeError, KeyError) as e:
+            rospy.logerr(f"Failed to parse /files response: {e}")
+
+    def _on_download_scan(self, msg):
+        thread = threading.Thread(
+            target=self._download_scan, args=(msg.data,), daemon=True
+        )
+        thread.start()
+
+    def _download_scan(self, scan_name):
+        """Downloads the specified scans from the Hovermap.
+
+        Hovermap will abort returning a HTTP 404 error if a scan is running, or
+        if you are already downloading a scan, so we don't need guards
+        """
+        url = f"{self._base_url}/downloadscan?scanname={scan_name}"
+        output_path = os.path.join(DOWNLOAD_DIR, f"{scan_name}.zip")
+        rospy.loginfo(f"Downloading scan '{scan_name}' to {output_path}")
+        result_msg = Bool()
+        try:
+            req = urllib.request.Request(url)
+            with urllib.request.urlopen(
+                req, timeout=SCAN_DOWNLOAD_PREP_TIMEOUT
+            ) as response:
+                rospy.loginfo("Zip file generation completed, starting download...")
+                total_size = int(response.headers.get("Content-Length", 0))
+                downloaded = 0
+                last_logged_pct = 0
+                with open(output_path, "wb") as f:
+                    while True:
+                        chunk = response.read(1048576)
+                        if not chunk:
+                            break
+                        f.write(chunk)
+
+                        # Progress messages
+                        downloaded += len(chunk)
+                        if total_size > 0:
+                            pct = (downloaded * 100) // total_size
+                            if pct >= last_logged_pct + 10:
+                                last_logged_pct = pct - (pct % 10)
+                                rospy.loginfo(
+                                    f"Download '{scan_name}': {last_logged_pct}%"
+                                )
+            rospy.loginfo(f"Download complete: {output_path}")
+            result_msg.data = True
+            self._scan_download_successful_pub.publish(result_msg)
+        except urllib.error.HTTPError as e:
+            rospy.logerr(f"Download failed for '{scan_name}': HTTP {e.code} {e.reason}")
+            self._scan_download_successful_pub.publish(result_msg)
+        except urllib.error.URLError as e:
+            rospy.logerr(f"Download failed for '{scan_name}': {e.reason}")
+            self._scan_download_successful_pub.publish(result_msg)
+
+    def _on_set_scan_prefix(self, msg):
+        self._get(f"/setprefix?prefix={msg.data}")
+
+    def _poll_status(self, event):
+        body = self._get("/status", log=False)
+        if body is None:
+            return
+        try:
+            data = json.loads(body)
+            status = HovermapStatus()
+            status.scan_prefix = data["scan_name"]
+            status.current_scan_name = data["scan_dir"]
+            status.free_space = data["freeSpace"]
+            status.scan_running = data["state"] not in NON_RUNNING_STATES
+            self._status_pub.publish(status)
+        except (json.JSONDecodeError, KeyError) as e:
+            rospy.logerr(f"Failed to parse /status response: {e}")
+
+    def spin(self):
+        rospy.spin()
+
+
+if __name__ == "__main__":
+    node = HttpInterfaceNode()
+    node.spin()

src/hovermap_api_msgs/CMakeLists.txt

@@ -0,0 +1,24 @@
+cmake_minimum_required(VERSION 3.16)
+project(hovermap_api_msgs)
+
+find_package(catkin REQUIRED COMPONENTS
+  message_generation
+  std_msgs
+)
+
+add_message_files(FILES
+  ScanInformation.msg
+  ScanInformationList.msg
+  HovermapStatus.msg
+)
+
+generate_messages(
+  DEPENDENCIES
+  std_msgs
+)
+
+catkin_package(
+  CATKIN_DEPENDS
+  message_runtime
+  std_msgs
+)

src/hovermap_api_msgs/msg/HovermapStatus.msg

@@ -0,0 +1,4 @@
+string scan_prefix
+string current_scan_name
+string free_space
+bool scan_running

src/hovermap_api_msgs/msg/ScanInformation.msg

@@ -0,0 +1,2 @@
+string name
+int64 size  # in bytes

src/hovermap_api_msgs/msg/ScanInformationList.msg

@@ -0,0 +1 @@
+ScanInformation[] scans