Orthorectified building facade textures from Mapillary street-level imagery and OpenStreetMap footprints. CPU only, no ML.
Left to right: the raw cut from the panorama, the occlusion mask, the fused texture, the same wall at 1 pixel per metre, and the wall/window/door classes. Munich, OSM way 81190155, three views fused.
Give it a bounding box. It fetches the OSM buildings and every Mapillary image in the area, works out which photos can see which wall, and writes one ortho-rectified texture per wall, plus a 1 px = 1 m version with each cell labelled wall, window or door. It also writes an inspection sheet per building so you can check the result instead of trusting it.
There is no object detection anywhere. It runs backwards from the geometry.
1. The wall is already known in 3D. An OSM footprint edge plus a height is a rectangle in space. For every point on that rectangle, form the ray to the camera, convert it to a direction, and read the pixel off the 360 degree sphere. Driving the loop from the output means perspective undoes itself: you get a fronto-parallel image at true metric scale, with no homography and no vanishing-point estimation.
2. The geometry gets corrected before it is used. Mapillary ships an OpenSfM point
cloud (~44k points) with each photo cluster. Points near a wall give the real plane,
since OSM outlines are off by 0.5 to 2 m, and the top of them gives the eave height,
which is usually better than the building:levels tag. Each photo's cloud is registered
against the OSM outlines first, because the two drift apart by a few metres.
3. Several photos, median-combined. Up to three views per wall. Parked cars and pedestrians move between captures, the wall does not, so they drop out of the median.
Every building gets a sheet. This is the top panel for the wall above:
Yellow dashed is where the raw OSM footprint lands using the raw camera pose. Orange is after registering the SfM cloud against the footprints. Green is the final rectangle on the fitted plane, with its 20 x 21 block grid drawn in. Green dots are the cloud points the plane was fitted to, red dots are detected occluders. The 4 m gap between the yellow and green outlines is the error this pipeline exists to remove.
The rest of the sheet carries the loose crop with its detected lines, the fused texture, the block grid and classes, a native-resolution inset, a plan view of the plane fit, the per-view textures with their residuals, and a table of every quality factor and every rejected candidate photo with the reason.
pip install numpy opencv-python scipy shapely pyproj Pillow requests matplotlib
python prefetch.py --bbox 48.135635,11.578243,48.137225,11.580818 --all-panos
python run.py --run munich --bbox 48.135635,11.578243,48.137225,11.580818 --stage all --workers 8
Then open out/munich/sheets/index.html.
Set MAPILLARY_TOKEN to your own token from
mapillary.com/developer. The default is
Mapillary's public demo token and is rate-limited. On Windows, run
set PYTHONIOENCODING=utf-8 first, and quote the token with double quotes in cmd,
since it contains |.
Stages are cached. Re-run one with --force texture --ways 81190155.
Per wall, in out/<run>/texture/:
| file | contents |
|---|---|
blocks/<wall>.png |
1 px = 1 m. RGB is the facade colour, alpha is the class: 255 wall, 192 window, 128 door, 64 unknown, 0 no data |
blocks/<wall>_cls.png |
the class channel on its own |
tex/<wall>_tex.png |
8 px per metre, for looking at |
facades/<building>.json |
wall geometry, OSM node ids, heights and their sources, per-view poses, confidence, and every flag |
Column 0 sits at the first OSM node of the edge, row 0 at the top.
Numbers from the Munich test box (48.1356,11.5782 to 48.1372,11.5808): 288 panoramas, 174 buildings, 1030 walls, 278 s end to end on 16 CPU threads. About 3.4 s per wall, 1.2 s per wall-view.
- 90 walls textured. Most of the other 930 have no camera facing them at all, being courtyard and party walls. That is a property of the street, not a bug.
- Rectification is solid. Vertical edges in the final textures sit within 1 degree of vertical on 27 of 28 measured walls, median 0.2 degrees. Plane fits are 0.15 m RMS.
- Heights beat the tags. Cloud-derived eave height matches
levels * 3to a median of 0.16 m. The sky-derived roofline sits about 1.9 m above it, which is the roof. - Windows found on all 90 walls (median 24% of blocks), doors on 89.
Honest about the rest:
- Registration is accepted on 34% of photos. The others fall back to a coarser cluster-wide shift.
- Multi-view agreement is within 0.75 m on 40% of walls. Not good enough yet.
- Window periodicity, meaning snapping to a regular grid, only fires on 11 of 90.
- The confidence tiers barely discriminate: 90 A, 1 B, 9 C. The scoring needs work.
Trees are detected, not silently averaged in. Vegetation is masked by a green-ratio and texture rule, closer geometry by the SfM depth map, and other buildings by ray-casting against their footprints.
A tree covering 46% of the wall. It is masked (magenta), excluded from the median, and what no view could see stays black rather than being invented.
Extra views help with cars and people. They do not help with a tree, because the tree is
in the same place in every photo of that sequence. Heavily occluded walls come back with
a high unknown share so you can decide what to do with them.
Both cost time to find out, so they are written down here.
The equirectangular images are not levelled. The pixels are as uploaded and the rig
tilt is still in them, so you must apply computed_rotation (axis-angle, world to
camera, ENU world, camera x right / y down / z forward; the rows of the matrix are the
camera axes). Skipping it cost 7.5 degrees of standard deviation in vertical-edge lean
in Munich, and 99 versus 38 of 128 walls within 2 degrees in Berlin.
computed_geometry is already the SfM camera position, agreeing with the cluster
shot centre to under a metre. Residual misalignment is OSM-versus-imagery drift, not GPS
error, so refining the camera position is wasted effort while registering the footprints
is not.
Also worth knowing: bbox searches must be under 0.01 degrees, and dense areas answer
HTTP 500 "reduce the amount of data" even inside that limit. Subdivide and retry.
Research prototype. It produces useful output and the failure modes are measured and visible in the sheets, but the numbers above come from one city. Berlin has only been used to check the rotation finding.
DESIGN.md has the full pipeline and module interfaces, CRITIQUE.md the corrections
that override it.
Code is Apache-2.0.
Output is a different matter. Mapillary imagery is CC BY-SA 4.0 and OpenStreetMap data is ODbL. Textures produced by this tool are derived from both, so attribution and share-alike apply to what you do with them. Check before shipping the pixels anywhere.
Mapillary for the imagery, the API and OpenSfM. OpenStreetMap contributors for the footprints. Using a coarse building model as the rectification target follows Texture2LoD3 (CVPRW 2025). The view-selection gates follow OpenFACADES.