From: Francois Gouget Subject: Re: RFC: Allow marking unreliable tests as flaky. Message-Id: <28bdd3e2-98bf-3170-d0cf-f3732029c585@free.fr> Date: Mon, 6 Jun 2022 16:23:43 +0200 (CEST) In-Reply-To: References: On Wed, 1 Jun 2022, Esme Povirk (she/they) wrote: > It might be better to use something like broken() to encourage more > precision in marking the failures. For example, if something fails > with ACCESS_DENIED sometimes, we'd still like to know if it starts > failing with FILE_NOT_FOUND. So something like this? ok(success || flaky(gle == ACCESS_DENIED), "Call failed: gle=%lu\n", gle); I suspect we'd get a lot of "|| flaky(1)", for instance for all the "always new" error messages. Also the goal is to make the tests more "reliable" but if we are too specific about which unexpected conditions we allow there's a risk we'll spend our time expanding the condition inside the flaky() call. But maybe it would be okay to give it a few tries and fall back to flaky(1). -- Francois Gouget