From: "Changsheng Chen" Subject: Re: [PATCH] d2d1: Fix the stack overflow error caused by d2d_cdt_triangulate(). Message-Id: Date: Tue, 31 Aug 2021 05:25:23 +0000 In-Reply-To: References: Hello, Henri Verbeet. Yes, it is at least greater than or equal to three, but it happens in the upper function d2d_path_geometry_triangulate(). Through some testing methods, I locate it before d2d_path_geometr_triangulate() calls d2d_cdt_triangulate(), it will change the value of vertex_count , Make it less than 3. When d2d_cdt_triangulate() is actually received, the value of vertex_count will cause a stack overflow error. In d2d_path_geometry_triangulate(), the code block to change the size of vertex_count is as follows: for (i = 1; i
Hello, Henri Verbeet. Yes, it is at least greater than or equal to three, but it happens in the upper function d2d_path_geometry_triangulate(). Through some testing methods, I locate it before d2d_path_geometr_triangulate() calls d2d_cdt_triangulate(), it will change the value of vertex_count , Make it less than 3. When d2d_cdt_triangulate() is actually received, the value of vertex_count will cause a stack overflow error.
In d2d_path_geometry_triangulate(), the code block to change the size of vertex_count is as follows:

for (i = 1; i <vertex_count; ++i)
     {
         if (!memcmp(&vertices[i-1], &vertices[i], sizeof(*vertices)))
         {
             --vertex_count;
             memmove(&vertices[i], &vertices[i + 1], (vertex_count-i) * sizeof(*vertices));
             --i;
         }
     }

We can see that the value of vertex_count has been reduced. At the same time, we saw in the test that after passing in d2d_cdt_triangulate(), the vertex_count is less than 3.
 
Of course, under normal circumstances, it is difficult to test and find this problem. I found that it was a drawing application that I migrated. Whenever I draw with the pen, I click the mouse habitually, and the application gets stuck. The debug log throws a stack overflow error. By tracking this problem, I found it and found a solution to it, which is through this patch of mine.
 
 
------------------ Original ------------------
Date:  Mon, Aug 30, 2021 01:09 PM
To:  "陈长胜"<chenchangsheng@uniontech.com>;
Cc:  "wine-devel"<wine-devel@winehq.org>;
Subject:  Re: [PATCH] d2d1: Fix the stack overflow error caused by d2d_cdt_triangulate().
 
On Mon, 30 Aug 2021 at 09:48, 陈长胜 <chenchangsheng@uniontech.com> wrote:
>
> When vertex_count==1 or 0, d2d_cdt_triangulate will always be called by itself, causing stack overflow error.

How does it end up getting called like that, do you have a test?

If I had to guess, I suppose it's possible that we end up with less
than three vertices after eliminating duplicates in
d2d_path_geometry_triangulate(). In that case though, it would make
more sense to move the vertex count check in that function after the
duplicate elimination.