Skip to content

Shape Healing - Guard ShapeFix_Face::Perform against non-face context replacement#1323

Draft
gsdali wants to merge 1 commit into
Open-Cascade-SAS:masterfrom
gsdali:fix/shapefix-face-noncast-on-context-replacement
Draft

Shape Healing - Guard ShapeFix_Face::Perform against non-face context replacement#1323
gsdali wants to merge 1 commit into
Open-Cascade-SAS:masterfrom
gsdali:fix/shapefix-face-noncast-on-context-replacement

Conversation

@gsdali

@gsdali gsdali commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Fixes #1322.

Problem

ShapeFix_Face::Perform casts Context()->Apply(myFace) to TopoDS_Face via TopoDS::Face(S.EmptyCopied()) in three places, none of which check the shape type:

TopoDS_Shape S = myFace;
if (!Context().IsNull())
  S = Context()->Apply(myFace);
TopoDS_Shape emptyCopied = S.EmptyCopied();
TopoDS_Face  tmpFace     = TopoDS::Face(emptyCopied);   // unchecked

When a fix sharing the same ShapeBuild_ReShape context has already replaced the face with a compound (e.g. a self-intersecting face split into several faces), Apply() returns a non-face. The unchecked cast then builds an invalid TopoDS_Face handle over a compound TShape; a subsequent operation corrupts the shape and the process aborts with an uncatchable OS signal:

BRep_TEdge::EmptyCopy() const
BRep_Tool::Curve(TopoDS_Edge, ...)
ShapeAnalysis_Edge::Curve3d(...)
ShapeFix_Face::FixOrientation(...)
ShapeFix_Face::Perform(...)

The fault address varies run to run (SIGSEGV / SIGBUS) - a heap-corruption signature. Diagnosed by linking a debug-built ShapeFix_Face.cxx: at -O0 the cast throws Standard_TypeMismatch at the first site with Apply(myFace).ShapeType() == TopAbs_COMPOUND while myFace is still a TopAbs_FACE.

Fix

The compound replacement already exists on entry to Perform (it was recorded by a prior face's fix), so a single guard at the top of Perform covers all three cast sites: if Context()->Apply(myFace) is not a face, record it as the result and return - the replacement is already in the context, so there is nothing to fix here.

Reproducer / test

Extrude a 4-point "bowtie" face (self-intersecting wire) into a prism and run ShapeFix_Shape:

BRepBuilderAPI_MakePolygon mp;
mp.Add(gp_Pnt(0,0,0)); mp.Add(gp_Pnt(1,1,0)); mp.Add(gp_Pnt(1,0,0)); mp.Add(gp_Pnt(0,1,0)); mp.Close();
BRepBuilderAPI_MakeFace mf(mp.Wire(), Standard_True);
BRepPrimAPI_MakePrism prism(mf.Face(), gp_Vec(0,0,1)); prism.Build();
Handle(ShapeFix_Shape) sfs = new ShapeFix_Shape(prism.Shape()); sfs->Perform();  // crash on master

Crashes on current master; fixed here. Added GTest ShapeFix_ShapeTest.HealPrismFromSelfIntersectingFace.

Validation

Verified by compiling the patched ShapeFix_Face.cxx and linking it over the library: the bowtie reproducer and four real mesh-derived fixtures survive (5/5), and the prism now heals to a valid 8-face solid. ShapeFix_Shape output on valid box/sphere/cylinder is unchanged (the guard only triggers for a non-face replacement, which never happens for a well-formed face).

Reported downstream and isolated at SecondMouseAU/OCCTSwift#263.

Draft pending CLA confirmation and CI.

… replacement

When a prior fix in the shared ReShape context replaced the face with a non-face
(e.g. a self-intersecting face split into a compound), Context()->Apply(myFace)
returns a non-face and the TopoDS::Face casts in Perform build an invalid handle,
corrupting the shape and crashing. Skip Perform in that case.

Added GTest ShapeFix_ShapeTest.HealPrismFromSelfIntersectingFace.
@gsdali gsdali force-pushed the fix/shapefix-face-noncast-on-context-replacement branch from e7cee19 to 5ee3535 Compare June 25, 2026 21:49
@dpasukhi dpasukhi self-requested a review June 27, 2026 16:56
@dpasukhi dpasukhi added 1. Modeling Boolean operations, offsets, primitives, any conversion, brep builders and etc... 3. CLA approved User has the signed CLA and ready to review or integration processes labels Jun 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1. Modeling Boolean operations, offsets, primitives, any conversion, brep builders and etc... 3. CLA approved User has the signed CLA and ready to review or integration processes

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

ShapeFix_Shape heap-corrupts (SIGSEGV/SIGBUS) when healing a prism extruded from a self-intersecting face

2 participants