EAS-202
===

Comparing Swift Testing and XCTest
---

[00:00:00]

Leo Dion (host): Welcome to another episode of Empower Apps. I'm your host, Leo Dion. Today I'm joined by Rachel Brindle. Rachel, thank you so much for taking time to come on

Rachel Brindle (guest): Yeah, ~happy to be here.~

Leo Dion (host): before we get into Swift Testing and all the new stuff with testing. ~Uh, ~I'll let you go ahead and introduce yourself.

Rachel Brindle (guest): Yeah, so I am Rachel. I am a principal engineer at Autodesk. And I have been involved with testing, test-driven development and such, basically ~my, ~my entire career. ~Um, so. ~A lot of fairly deep knowledge of like how to go through and test things, as well as mentoring others in learning ~how to ~how to test code, ~how to, like, how to deal with some of the, uh, ~how to write better tests that aren't brittle and won't break all the time and so forth.

Leo Dion (host): Awesome. So what year did Swift Testing come out? ~Was that last year or a~

Rachel Brindle (guest): That was, ~uh. ~It shipped in Xcode 16 last year, but ~it was like an open, ~it was open sourced in September, 2023.[00:01:00]

Leo Dion (host): Okay. ~Yeah, yeah. ~So ~you, ~you did a lot of work with XCTesting. You have a few libraries you actually built, ~uh, ~for XCTesting, ~uh, ~to help folks. What did you think about the process as far as Swift Testing is concerned and how we got to where we are now? How we went from XCTesting to Swift Testing. And what do you miss? ~What do you, ~what are you happy that we got rid of Things like that,

Rachel Brindle (guest): ~yeah. Okay. Yeah. So, uh, in term. In terms of, uh, what I miss. ~So Swift Testing is obviously, 'cause it's only like publicly, it's only like a year and a half old, though I'm, I know for a fact that like, it didn't just pop outta nowhere. ~Uh, ~they've been working on it since at least macros were a thing. Swift Testing, if it's still like no more than, it can't be older than like three years.

It is. Nowhere near as mature as XCTest is. XCTest has been, was introduced in iOS seven, iOS eight era. But even still it dates back to much [00:02:00] older Objective C focused ~a, uh, ~testing APIs like syn testing kit and so forth. And those date back all the way to the nineties.

Leo Dion (host): I was gonna ask about that. 'cause like XCTest, I would assume. Like you said is from quite a while since Objective C. 'cause I can't even imagine what they used for testing before that.

Rachel Brindle (guest): Yeah, it was, ~um, ~I don't think Apple had, ~uh, ~so it was initially started as some, ~um, ~third party, like predating. It might even predate like Apple buying next. I don't that. That I, I don't quite know the history there. But it was, there was a send testing kit and like a bunch of like other third party stuff is what people used.

At some point in the two thousands, apple started shipping or like including send testing kit with, ~uh, ~Xcode so that you didn't have to download that, ~uh, that as a third party thing. ~And they started to just refine that over time and eventually they rewrote it with XC test and, and shipped that.[00:03:00]

Leo Dion (host): Okay, so Swift tests, like the big thing I was gonna talk about is what you miss from XCTesting, but then like a part of that is just what we could do with Objective C that we can't get away with in Swift.

Rachel Brindle (guest): ~Somewhat. ~So there is the open source XCTest. ~There's the, ~it's Swift CoreLibs XCTest and that is written entirely in Swift and that enables that has been around ~since the, uh, ~since Swift package manager was open sourced back in like 2015 or so. And ~that in a, ~that is not feature complete ~with xe, ~with like the private XCTest from Apple.

But ~the, ~it does a lot of what you can do with XCTest. ~That enables a lot of the like dynamicism of it. ~XCTest is really ~much more mature for, or ~much better for ~like ~building testing tools on top of, like you can dynamically create and register tests at runtime and XC test, which is not a thing you can do in Swift Testing at all. I know that there are people on the Swift Testing team interested in enabling something like that, but it's just not there yet. ~There's other stuff like, um, ex like automation tests are built on top of. Test. ~Though [00:04:00] reading between the lines, they're definitely moving to enable that feature in, ~uh, ~Swift Testing.

And then there's other minor stuff like, ~uh, ~performance tests. The XCTest has the measure API, so you can ~go and, um. ~Figure out how long it takes to run a bit of code so you can see like, oh, do I have any performance regressions in this area and such? And you can kind of do that with Swift Testing, but ~like, ~for a bunch of reasons, Swift Testing's, concurrent test runner being a major issue there.

~Uh, ~it's not great. And ~the, ~of course there's not really any equivalent to ~like ~the measure API. ~Yeah.~

Leo Dion (host): So ~what, ~what did you really like about Swift Testing and where it's headed right now compared to XCTesting? ~I.~

Rachel Brindle (guest): So the thing that I really like about it first, the DSL of the experience of actually writing tests is much nicer than with XCTest. The macro or just being able to specify like the attest macro and then just like a bear function, ~uh, ~is great, [00:05:00] especially in Swift six two. I think it's se 4 51 ads just like you can, I think it's bear function.

~It's like bear identifiers. ~You can use ~like ~a backtick ~to, ~and then put ~just like ~an arbitrary string or nearly arbitrary string between back ticks and like that's the name of your function. So you can have ~like ~spaces commas and ~like ~punctuation ~and such ~in there. ~And so ~that will ~then like ~render nicely as ~like ~a full human readable string so you don't have to do this weird thing where like you have attest and then whatever human readable string and then a non-human readable, but like valid function identifier afterwards.

Leo Dion (host): Okay, so now ~you don't, ~you could just make the function whatever you want, and put whatever characters you

Rachel Brindle (guest): yeah. ~Yeah.~

Leo Dion (host): That's crazy.

Rachel Brindle (guest): ~so much better. ~That's useful for non-testing stuff, but mostly it's useful for testing. That's like reading ~the, uh, uh, for ~the proposal for that. That's like the main motivation they cite.

Leo Dion (host): Yeah. ~Makes sense.~

Rachel Brindle (guest): Yeah. So yeah, like the, going back to other things that I really like about, ~uh, ~Swift Testing I like how it's much more [00:06:00] open. Both like it's developed mostly in the open. ~There is like the, uh, integration stuff with Xcode that is necessarily private and such, but the ~there is a testing work group, which I'm a member of.

And it's. It, ~uh, ~follows a form of the Swift evolution process. Like testing proposals are in the Swift evolution repo. But the proposals are formatted slightly differently and such so I'm really liking that there's a lot more community engagement in the development of Swift Testing.

Leo Dion (host): Yeah. And I think that's the biggest advantage is the fact that it's open source and that we can ~like, ~build a whole set of tools around that and ~help, ~help it facilitate its growth, ~what, at ~what point do you think it'd be important for a team to migrate from XCTesting to Swift Testing?

Rachel Brindle (guest): Really that depends on the team, there's a ton ~of, there's a bunch of ~of tests that are written right now that cannot be easily migrated or like can't, it's not gonna be an easy process to migrate it from XCTest to Swift Testing.

Leo Dion (host): Do you have an example?

Rachel Brindle (guest): So like anything that accesses like, or [00:07:00] modify some like shared global state, ~like if you have like a global, like variable that's actually mutable and you might like.~

For like one specific test, you might change it to some value. And then another test wants it to be ~some other va ~some other value. ~Uh, ~because Swift Testing runs everything concurrently in process, then you're gonna run into issues if those two tests are ever actually like running at the same time.

Leo Dion (host): Should you be doing that in the first

Rachel Brindle (guest): I mean no, ~but like plenty of people do that.~

Leo Dion (host): ~right. Right. Okay. I wanted make sure. ~Okay.

Rachel Brindle (guest): Yeah. Yeah. ~I'm like, it's. ~In a way it forces you to be cleaner. Like you'll end up replacing that type of thing with a task local or whatever. And that also brings along all the niceties of like, you're now going to force your tests to actually like build under the Swift six language mode and all that such.

But there are plenty of tests that exist right now that work perfectly fine in the world with XCTest where it's only one test is running at a time in the process. That are going to be difficult to migrate to Swift Testing.

Leo Dion (host): Okay.

Rachel Brindle (guest): And then ~like, really the o ~the other benefit is ~like, ~it'll be ~a ~slightly easier to read stuff.

You'll get faster runtime of tests because they can execute in [00:08:00] parallel but ~like, ~you're gonna get slightly slower build times because macros take a bit of time. Even ~like ~tool chain macros, ~uh, they just, ~they just add a little bit more time to the compile time. ~Um, ~automation tests still aren't a thing and so forth.

But yeah, ~like ~if you have working XC tests, there's no reason to, ~uh, ~unless you really want to, there's no reason to go rewrite them into Swift Testing.

Leo Dion (host): ~Would you write new tests in Swift Testing? ~Maybe.

Rachel Brindle (guest): So like that's the approach to that idea is ~like ~new components are written in Swift Testing. But ~like ~if it's an existing component that already has tests written for it, then ~I am, ~I'm just gonna keep it in XC test.

Leo Dion (host): ~great. The big thing that kind of unlocks Swift Testing with macros. ~Do you think macros have a place in filling the gap when it comes to what's missing? In Swift Testing?

Rachel Brindle (guest): Yes and no. So well for I guess to correctly answer, yes, they have a gap. But like they are very painful to write and maintain as I'm sure you don't need to be reminded, ~uh, ~I looked at how, ~uh, ~the expect and require macros and Swift [00:09:00] Testing are implemented and like mad respect to the people on the testing team who wrote them, because I do not know if I could.

And there's also the other stuff of like, until very recently you had to rebuild sw, ~uh, ~Swift, Swift syntax for like. Third party macros and such, that isn't really a thing now that we have like the cast with Swift syntax as of like 16 four or code 16 four. But like there are, there actually are places for macros.

Every so often someone will post in ~like ~the Swift forums, the community showcase about a library that uses macros to generate test doubles. Conforming to ~like some, ~the ~like ~protocol specified. So ~like ~a test double is just kind of ~like ~a stand in for the real thing.

Leo Dion (host): Okay. ~Okay.~

Rachel Brindle (guest): So ~like ~you might have ~like ~your type that has ~like ~some methods or whatever and you have a protocol that defines that type.

You might then use one of these tools to. Create a test double that automatically [00:10:00] conforms to and implements ~like ~all of these methods such that you can then either just directly pass it in ~and it, ~and it'll work ~as it'll work like ~in the thing. Or you can ~like ~modify behavior or you could even ~like, ~examine how it was called and so forth.

~Um, and like ~that's really cool to see and that saves a bunch of boilerplate. Yeah, so ~that, ~that's ~like ~one of the main examples of ~like ~using macros to ~kind of ~fill in the gap and ~like ~do things that Apple probably isn't gonna do, at least not with Swift Testing.

Leo Dion (host): At least not in the next decade. Yeah, you had this like marketable macro, which is really cool. I just started looking at that, ~um, ~because like that's one of the things people love about Objective C is everything is dynamic so you can go in and swizzle and do all sorts of crazy stuff, whereas Swift is super strict about that stuff.

Rachel Brindle (guest): Like for good reason, ~like outside of ~like, outside of using tests to like mock out behavior, we might get the into this in a, in a little bit, but like this is part of what made testing UI kit ~like, uh, ~a lot less painful was being able to swizzle out a whole [00:11:00] bunch of stuff. Which you can't even do any of that with Swift test ~with, ~with Swift UI.

And like Swift UI is also not untestable.

SwiftUI Testing
---

Leo Dion (host): Yeah. Well, let's talk about that. That's one of the biggest complaints is how SwiftUI is testable. What does that mean exactly? ~Before we get into it?~

Rachel Brindle (guest): Yeah, so you can write tests for Swift UI ~for code using written, ~using Swift UI ~you, ~but you can only write automation tests for it. So that's like the XCUI test stuff that, ~uh, it ~works by spinning up your app in the simulator and then having another process that actually runs the tests that. Uses IPC to interact with the app using the accessibility system.

Leo Dion (host): Yep.

Rachel Brindle (guest): ~Um, ~and out of the box you can only like, just interact with the UI and examine stuff in the ui. You can't like go in and examine oh, did this insert something into, did this actually insert something into the database or is the UI just lying to me? Out of the box, it's going to make like every single network request and so forth, which it's [00:12:00] not really.

~Is not great for like testability and such. ~But there's a bunch of third party stuff to make, ~uh, ~automation tests, ~uh, ~or like to fake out network requests and so forth and like enable you to better inspect stuff going on at automation tests.

Leo Dion (host): Yeah, well, like one thing ~I ~I hear a lot about is snapshot testing as a way, which I think is from the point free team.

Rachel Brindle (guest): so there are several ways you can do snapshot testing. ~And snapshot testing point the point. Free folks have a thing for doing that. ~There's an, ~a much ~older, ~uh, ~API from I think Uber was the first ~people ~who made it ~Uber, Facebook, one of those groups. That's, uh, ~iOS snapshot test case ~that's.~

~But basically ~both of these, the 0.31 does a little bit more, you have a little bit more options to ~like ~actually inspect it. ~I'm more familiar with the other one. Um, ~it works by just rendering the UI into an image and then basically ~working or ~working entirely as a regression test of like, you render the eye into an image.

You test that and you just assert that. The image is the same as ~like ~a previously recorded image ~was.~

Leo Dion (host): Yep.

Rachel Brindle (guest): That can work either as like, like an XCUI automation test or as more of a unit test, like in process [00:13:00] test. And that's fine. Like it works for like pure regression stuff. There's, it's very brittle.

Like I do not envy the people with, with snapshot tests, ~uh, ~going into who are probably not having a very fun summer, having to rerecord all of their stuff for like the liquid glass~ uh, ~design, language change. And especially like each new beta, it's gonna change. But even going between like minor versions of the os, there, there are like gonna be slight changes to how things render and so forth.

There's even like stuff of ~like, ~whether when the simulator runs on like an Intel or like a, ~uh, ~apple, silicon Mac you, it will render it slightly different. So you have to do like some sort of like fuzzy image compare instead of just doing a straight up,

Leo Dion (host): You're right.

Rachel Brindle (guest): pixel by pixel compare.

Leo Dion (host): Okay.

Rachel Brindle (guest): so like snapshot tests work, but they're extremely brittle. When I say that Swift UI [00:14:00] isn't testable, what I mean is that I can't write ~a, ~a unit test for it. One of the things that you can do with UI kit is you can directly call the methods that Youi kit itself calls when it's like when you tap for that lead to ~like ~a button callback being called. So it's possible for me ~to like write code and or ~to write a test that's just like, Hey, when this button is tapped~ the, ~I write to the database and I can actually then inspect that the database was written to, or ~like ~that a test double that represents the database was talked to. ~Um, ~and that's not something you can do ~at, on, uh, ~with Swift UI.

You can't simulate a button. Press in test.

Leo Dion (host): ~Could you, if you're a glutton for punishment, ~could you use something like an inspect and push the button?

Rachel Brindle (guest): Yes, ~there is a, uh, so ~there is a third party library called a view Inspector. ~I'll add it, ~I'll add it to your, ~uh, ~show notes~ and that. ~To its credit. It does a really good job of allowing you to inspect and test like, Swift UI stuff, ~uh, ~Swift UI views and so forth. But it's incomplete. ~It, there's a bunch of like app, uh, ~there's a bunch of APIs from [00:15:00] Apple that it doesn't have good inspection for.

~'cause it has to use, ~it uses the mirroring API to be able to inspect things. And then it has to know about specific. Like sub components within to, or like to go like inspect, oh, this was the, ~uh, ~UI button or the, sorry, the Swift UI button. ~Like ~tap handler and such. It doesn't ~just ~automatically support new Apple provided, ~uh, ~Swift UI views by default.

It does work with like your own third party views 'cause. ~Then it can, you can directly inspect like the properties on it because of the whole access control stuff.~

But it works. It's also brittle, it's vulnerable to internal changes to in Swift UI of

Leo Dion (host): That's exactly what I was gonna say ~is like if Apple changes something, good luck.~

Rachel Brindle (guest): that happened last year. So, yeah. ~Uh, ~really what I would love to see is first party support for that. I know for a fact that ~like ~every time, 'cause I bring it up every year and a WW TC lab that both the Swift UI and testing teams get that very frequently as a request. But for various reasons they [00:16:00] haven't been able to deliver that.

Leo Dion (host): How would you picture something like that? Like you just create a Swift to eye view and then some sort of select, like ~how would Web, ~how does web do it? Right? Web would just be like, oh, grab the button using a selector, and then push. ~Push the button. Is that what you're~

Rachel Brindle (guest): ~Yeah. ~Yeah. So that's exactly what I'm thinking. Like, something that I've been working on as like a blog post is, ~um, ~exploring what that might look like. Through the lens of like, ~well ~this is what we might try to do for UI kit and then because we can do it for UI kit. I like the approach, the interacting through the accessibility system that automation tests take.

'Cause that already exists, that forces you to actually add accessibility, support,

Leo Dion (host): I was gonna say that.

Rachel Brindle (guest): And such. ~Uh, ~I think that'd be a really good thing to do. ~Um, ~plus it's also fairly, like black box, like you could call ~in a, ~in an automation test, you can call ~like ~a tap on an element that cannot be tapped. It's [00:17:00] non-interactive and nothing will happen.

Like you won't get a compile error. It's just that nothing will happen and your test will silently fail. ~And ~or maybe it won't fail depending on what your next assert is~ or won't. Yeah. The, ~that's something that I think is. Really powerful as ~like ~a direction to go ~to ~maybe ~not exactly mirroring or definitely ~not exactly mirroring the, ~uh, ~public API of automation tests.

But that's a great starting point of let's interact through the accessibility system to be able to like pull on, pull an element that should exist. ~But do, uh, ~and then like be able to interact with it, test or tap on it, swipe on it, and so forth.

Leo Dion (host): Yeah, that makes complete sense.

Advanced Testing Techniques and Tips
---

Leo Dion (host): So I wanna jump back, talk a little bit more about Swift Testing. So one of the things I really liked about it is the way that you can organize your tests, ~um, ~without needing to do certain things with classes and stuff. Kind of explain the whole suite testing thing and what are some things developers should be thinking about when they write their tests and how they organize them.

Rachel Brindle (guest): [00:18:00] Yeah, one of the really cool things about Swift Testing is the fact that you can nest sweets as much as you want. ~Uh, ~you can have zero sweets, so just bare functions that are with that test macro. You can organize, you can have like a single layer of sweets. You can have as many as you want.

There is, you do kind of ~run, you do ~run into the issue where, ~uh, ~if you have properties on like the parent suite, a child suite cannot reference it because they're structs and they can't know about, they don't, they're not getting like an a reference to like an instance of the parent struct and so forth.

So you can't you have that limitation. I'm hoping that once apple. Provides like more APIs for like dynamically ~adding, for ~adding tests and so forth, dynamically that there'll be community solutions to that. Something like what the quick framework provides for XCTest.

Leo Dion (host): Could you give ~like an example, ~like a use case of that?[00:19:00]

Rachel Brindle (guest): so

Leo Dion (host): a parent suite info or metadata.

Rachel Brindle (guest): Yeah, yeah. So that, that's something like, ~um. ~Even just as simple as like having shared setups. So you only have to create the object being tested like in one place in the, like in it for the, in like the parent or whatever, the outermost suite. And then later on, like you might have like a child suite that is just calling a method on that object that's being tested.

That's. One way to do that. I also like to nest suites, or I would like to nest suites for like, conditional and such, a child, one suite for if this conditional is true. Another for if it's false and such. Because then you also get like the nice fo code, code voting capabilities~ so that you can.~

~Easily. Easily just say like, ~well, I don't really care about the positive case right now, so I'm just gonna fold that up and only look at the negative case.

Leo Dion (host): Like what,

what I use a lot, and we'll get into traits later, but ~is like ~I have to have, I'll have a suite [00:20:00] that I do a lot of full stacks with stuff and so I'll have ~a, ~something that's only supported on Apple platform, so I end up having to do like pound if can import or something like that to kind of distinguish between the two ~and or.~

SOI, for instance. And then ~like ~I have to create a variable and then use that as ~like a, basically ~a global variable that says whether it's available or not. ~And then use to say based on that, ~which it's kind of a mess, but it works. We'll talk about traits more but back to like sweets. What do you think, ~um, ~what are some tips you have as far as how you should organize your tests?

Rachel Brindle (guest): Yeah, I personally tend to just do one layer of sweets just because ~I, what I would like to do is ~I would like to nest them for conditionals, but doesn't really work out well. So I just do one layer and then either organizing them with tags or using. ~Just ~good old ~like ~Mark comments.

Leo Dion (host): ~Okay.~

Like I usually just do, ~I, ~I'm still stuck in the way of like. I'll have a test suite lined up with a type essentially and do [00:21:00] it that way. Is that, do you feel like that's still a good way to go about

Rachel Brindle (guest): ~that ~that's how I do it. ~Yeah. ~I'll have like ~a, ~a suite that's like, ~um, ~my object tests

Leo Dion (host): ~Yeah. ~Okay.

Rachel Brindle (guest): I'll

Leo Dion (host): And it's good for like code coverage too, because then you could tell, okay, what is this testing ~and then what are you~

Rachel Brindle (guest): Yeah. But you could also do that, ~like ~without the suite. It does make it nicer in terms of getting the, ~like, ~test diamond, ~uh, ~on the ~like ~left side of the editor because then you can just select everything there.

But ~you could also, ~if you didn't want that, ~you could. ~You could just have them all on top of the file and then have ~like ~the file be ~kind of ~your top level organization for it,

Leo Dion (host): Got it.

Rachel Brindle (guest): but then you don't get like shared setup and so forth.

Leo Dion (host): Right, exactly. ~Yeah, so I kind of talked a little bit about platform and os disabling using traits. ~What are some other traits that people should be aware of?

Rachel Brindle (guest): ~Yeah. So, um. You talked about like the enabled disabled traits for your case, actually, I just, ~I would recommend making like a custom trait that automatically does the, like, oh, if Mac, if iOS, if Linux or whatever,

Leo Dion (host): How hard is it to do that?

Rachel Brindle (guest): it's not that hard. ~Uh, ~you can, ~it's create a, uh, ~create a type that conforms to, you'll probably want to do both test trait and sweet [00:22:00] trait.

~Uh, ~and if you do conform to sweet trait, ~you will. ~You should implement the is recursive and then just have that be true is recursive means that for a sweet trait, it also applies that trait to every single like child of it, all the tests, all the sub suites and so forth.

Leo Dion (host): Yeah, that makes total sense.

~Could just call disabled in there basically.~

Rachel Brindle (guest): you can just have it say like, this also is a disabled trait, or like, this marks the test as disabled.

~I didn't quite look at how to do this prior to this. Of course. ~

Leo Dion (host): A fun project for me ~after this,~

Rachel Brindle (guest): yeah, but ~it, ~it shouldn't take that long to do.

Leo Dion (host): Okay.

Rachel Brindle (guest): There are the, ~uh, ~comment in bug traits, which just allow you to associate like text information. With a tester suite.

Leo Dion (host): Okay.

Rachel Brindle (guest): There is the serialized trait which forces the test covered by that suite or the like ~test if in the case of like parameter, a ~parameterized test to run ~like ~serially, so not concurrently,

Leo Dion (host): Okay.

Rachel Brindle (guest): ~that has.~

That's kind of weird. 'cause that only applies to like those specific tests. ~And in fact, ~if you have like two like sibling tests that both have like the serialized trait and you [00:23:00] run them, but like the parents, the containing suite isn't serialized. If you run those, then like those two tests will actually run concurrently or like in parallel with each other.

Leo Dion (host): Should you be doing serialized tests?

Rachel Brindle (guest): it's a thing that exists. ~I. Don't~

Leo Dion (host): Especially in a post Swift six world, like

Rachel Brindle (guest): I haven't run into like a specific use case that needs it. I am a hundred percent certain that that use case exists, otherwise it wouldn't exist.

Leo Dion (host): right.

Rachel Brindle (guest): But like every ~ting single ~time that I've used that I've ~like ~reached for it, it turns out that actually what I should have just done is made whatever the thing that was forcing the thing to be serial to actually make it.

Be thread safe, like change a global variable into a task local or something like that

Future of Swift Testing
---

Leo Dion (host): so I did want to cover your article, which I'll put in the show notes, but what are some highlights about, ~as far as like ~what's new this year that folks should really start looking at [00:24:00] using in their tests?

Rachel Brindle (guest): Okay, so the major new change is exit tests, which doesn't apply to iOS. ~That's kind of sad. But the.~

Leo Dion (host): on ~like ~the server side team?

Rachel Brindle (guest): Yeah, service side, even Mac development windows and such. They allow you to verify that. Code exits properly as you expect ~that it throws in or that like, it~

Leo Dion (host): code.

Rachel Brindle (guest): returns an exit code.

~It, you know, it calls exit with like, whatever the value you expect. ~It throws

Leo Dion (host): ~wow. ~Okay.

Rachel Brindle (guest): or something like that.

Leo Dion (host): So you can basically say, Hey, make sure the signal is the right integer. ~And then you can also take a look at standard error context, which~

Rachel Brindle (guest): Yeah, so you

Leo Dion (host): standard error context. What? How do you check that? ~That's what I'm trying to figure out that~

Rachel Brindle (guest): so,

Leo Dion (host): ~I see. ~The result has STA standard error content as a property.

Rachel Brindle (guest): ~yeah, yeah, yeah. Sorry. Yeah. The, ~the example I wrote could have been better,

Leo Dion (host): ~No, no, no. ~Now. 'cause I didn't understand what result was and I was like, I'm looking at this, and I'm like, okay. So result is basically some sort of structure, which has all that metadata

Rachel Brindle (guest): Yeah, so

Leo Dion (host): okay.

Rachel Brindle (guest): it has the, ~uh, ~if you tell it, if you tell ~your ~your exit test to grab the standard error, then it will. Then that, ~uh, ~standard error [00:25:00] content will be non nil. If you specify with standard output, it will also be a non nil thing. Those are just an array of, I think it's U eight but an array of, yeah, it's array of U eight.

It's array of bites that if you wanna, you then have to like convert it to a string using one of the methods there. If you wanna do a string comparison. ~Uh, ~there are like some weird beta bugs in there right now where it will like, capture a lot more of the like standard error than you need. And that's just kind of how the, ~like ~posik spawn, ~uh, ~API

Leo Dion (host): Oh, really? ~Okay.~

Rachel Brindle (guest): So yeah, it's. But ~it works ama, ~it works really well. I'm really impressed with how this works. ~The and ~you can't do this with XC test at all. Like Apple's never added support for this. So if you wanted to verify that your function had a precondition or such in it, and that precondition worked as you expected, you had to go way outta your way to like.

Construct something to do that, [00:26:00] which no one has ever done in their life.

Leo Dion (host): Yeah

Rachel Brindle (guest): so it's really nice to have this.

Leo Dion (host): that ~totally ~sounds like a server side thing.

Rachel Brindle (guest): Yeah, so really, I think I ~really ~want it to come to iOS. ~Um, but like obvious, like ~the sandbox in iOS is a thing and that's gonna be difficult.

Leo Dion (host): what's the use case for iOS? Because like if the, the app shouldn't crash, I mean, it should just never crash.

Rachel Brindle (guest): Right. But you do have APIs where it's like, Hey, I expect, for example, ~uh, ~array subscript, right? So, or like subscript an array. If you pass negative one then, or if you pass like an out of bounds index, then

Leo Dion (host): fatal error,

Rachel Brindle (guest): a fatal error. And like that. Whether or not it should or, or should return nil or do or like throw an error or something that you can actually handle is another question.

But that sort of thing exists and we should be able to like handle that test case just to have like that coverage that it's working.

Leo Dion (host): I was about to ask that. 'cause a thousand years ago when I did a, an episode with on this very [00:27:00] subject, I was asking like, how do you test for fatal error or precondition or assert or any of that stuff? And the way we've had to do it is basically have a way to like overload it essentially. ~Um. ~But we don't have anything like that.

And this, which what it sounds like is this would work in that case, but then at the same time it wouldn't work on iOS. Is that correct?

Rachel Brindle (guest): Yeah, so the way that I would approach this is I would, I would either just not, or I would like put in like an inject, like a, an injectable using like dependency, injecting like a shim that just in actuality it calls, as you were just saying, like it just calls precondition and actuality. And then in test we can examine that it was called that.

For this I might either write, like if it's a cross platform app, like I wanna have it run on Mac OS and iOS then I would just have the. Have it be covered under the test for macOS with like a pound if or like if os [00:28:00] macOS stuff covering that. ~And I don't know, maybe in the future they'll add this.~

There's a few things that are promising that isn't in there yet. The thing ~that, like, ~that I didn't write that I'm most ~excited for. 'cause there is, the thing that I, there is the pitch that I wrote that I am extremely ~excited for is a thing called, ~uh, ~issue handling traits. And these allow you to filter or ~like ~modify issues.

So like if you have a pound expect ~that is fail or ~that fails, like it. Pound expect true, equal, equal false. That's, guaranteed failure there. Then it allows you to, ~you can ~specify a trait that allows you ~to say, ~to modify the issue. You might wanna convert the, ~uh, ~severity of it. ~You might wanna change, ~you might wanna just entirely

Leo Dion (host): Yeah. Okay.

Rachel Brindle (guest): That's really powerful. ~I'm really excited for that to happen. ~I don't think it's gonna happen in Swiss six two.

Leo Dion (host): But you think it's possible in the next six ~or 3, 6, 4.~

Rachel Brindle (guest): Six probably six three. It's actively being discussed. Yeah,

there's other stuff there that's, ~uh, ~exciting, ~um, ~from what it enables, like, ~um, ~range confirmations, which [00:29:00] were merged in, ~uh, ~Swift six one and have been out since Xcode 16, three earlier this year. ~The, that is very much one of the, like, reading between the lines.~

This is a thing that ~is, ~they view as necessary for automation tests.

So range confirmations is basically just, you expect the callback to be called within some amount of range. And the example given ~in the ~in the proposal is using an automation test of like, oh, you have a button and. You can expect it to be tapped, clicked once, but what if ~like, ~the user actually interacts with the app at that point, like a human operator on the simulator or something like that.

Like in that case, you'll get a Furious tap event and it'll be or you'll get like ~an ~a tap event that the test wasn't expecting and then you'll get a failure that you don't really care for. There's also, in that same vein, attachments, ~uh, ~were added, I'm just jumping around in this article.

~Sorry. ~

Leo Dion (host): So is, is attachments like any sort of like metadata you want to attach to a test?

Rachel Brindle (guest): [00:30:00] Yeah. So this will be, this is already pretty useful to have as like. I can attach ~like, oh, I might have ~like a ~j uh, ~generated ~like ~JSON payload in the test. And then for debugging reasons, I want ~to, like debugging why a test failed. I want to be able ~to inspect what this JSON was so that I can more easily see like, oh, ~okay, ~this is actually an issue with my production code, or this is an issue with the test, or so forth.

But. They're also really useful in automation tests where they're used for storing~ for storing like oh my God, screen, uh, ~screenshots of what the app was doing, ~uh, ~at the time, or even ~like ~videos and such. ~The new ~I don't remember if the current code does this, but I know that ~the, like ~Xcode 26 supports recording video ~of.~

Like failing automation tests, which is really cool. ~So~

Leo Dion (host): Yeah.

Rachel Brindle (guest): That's a thing that ~you're gonna, ~is probably not necessary, but ~like, ~makes the experience of debugging automation tests ~doable and like ~much easier, especially when these things are running on ci. ~So I am, ~both of those are things that I'm really excited for in terms of what they enable.

~In the future, even if I'm not really gonna use them as much myself.~

Leo Dion (host): Let's talk [00:31:00] about it. Actually before we get into that, ~you wanna, ~you wanna talk a little bit more about specifically your proposal regarding polling confirmations?

Rachel Brindle (guest): Yeah. ~Uh, ~one of the libraries that I maintain is called Nimble. ~Nimble is a, I didn't write it. I just am the current maintainer. ~Nimble is a library that I believe, ~uh, ~Swift Testing took a lot of, or at least the pound expect and pound require. APIs took a lot of inspiration from, I haven't asked to confirm this but that provides a, syntax for, ~um, ~it's also inspired by ARS Spec. So, or Nibble is inspired by ARS Spec. So like there's that too. It provides just a syntax for that's better than the XCT assert, in my opinion. For comparing v. Asserting that some behavior happened, something happened. ~One of the things that, or ~one of the features of Nimble is something that I call pulling expectations, which is like the form is, ~uh, ~expect something to eventually something else.

So ~like expect one to eventually equal two and. The, I that's a bad example there, but like, ~the idea ~of this, ~of a polling [00:32:00] expectation is that it will run the thing that you expect. ~So like it'll run the ex, ~the value being sent into expect is actually ~an auto closure or is a, is ~a closure.

So it'll ~just ~run that closure~ many, ~many times pulling it continuously until it passes ~the, like what's called ~the matcher, in this case, equal two. So in this case, or in the example I gave, that would always fail, but if you gave it like a method and then that method might ~like return ~change, how its return value is over time then that might eventually pass.

This is something that like I think is really necessary for Swift Testing, ~uh, ~to support. You can. Kind of do it as a third party, but ~like, ~as I found it actually implementing this as a pitch there are problems with ~kind of ~the more naive solution or approach to this. So yeah, ~really the, ~in the article I give an example of like, all right, you call some method on ~a, on like ~a background task.

And then in the test you wanna say like, ~I, ~I want to. Confirm ~that this ~that this eventually, [00:33:00] you know, the method being run in the background ~t ~that it eventually ~like ~does its work. And that's precisely what this is doing. ~Uh, there are other~

Leo Dion (host): like in your example, it's like it keeps calling raised dolphins until the account gets up to one, ~basically. ~Is that correct?

Rachel Brindle (guest): ~uh, not quite. ~So what happens is that raised dolphins is called once but. As part of

Leo Dion (host): ~in the background 'cause it's a separate task. ~Okay.

Rachel Brindle (guest): yeah, as part of, and as part of like the internal implementation of it, it might increment the number of dolphins on it. And so then you're just checking like that is that eventually there is at least one dolphin there?

~Yeah.~

Leo Dion (host): I assume it would take some sort of time interval or like duration. ~Okay. ~'cause yeah. ~All right.~

Rachel Brindle (guest): Yeah. ~That's~

Leo Dion (host): really interesting,

Rachel Brindle (guest): yeah. That's been like a thing of, that's the, ~uh, ~time interval thing of like when to cut this off has been the most eye-opening thing about this

Leo Dion (host): right?

Rachel Brindle (guest): ~of like the way that concurrent systems run and also the way that like Swift Testings parallel execution runs. It.~

Swift Testing submits every single one of your tests to be run at once, and it submits it all to the Swift concurrency shared thread pool. [00:34:00] So on your system, you might have that shared thread pool, might have like 10 threads available, but if you have like 300 or so tests, you know, to run, ~then it still, it then like.~

Then it's gonna run those all and they might all be running concurrently, or at least all spend some amount of time on that, ~like ~shared thread pool before, ~like, and ~a wait causes the next one to run. And so that prevented an issue with the timeout. Or ~like ~the desire to ~have like, oh, ~have this automatically timeout after like one second.

If you have 300, 3000, you know, 10,000 or whatever tests to run on it. ~Um. ~And then your test gets run, you then like await to as part of this. There's no guarantee. And in fact, the more tests you have, the more likely, and especially in like resource constrained environments like ci there's no guarantee that you'll actually get that callback within that, like one second timeout.

So on like extremely large on extremely large test suites, this [00:35:00] will. Polling confirmations as I originally implemented it, I have fixed that now. Will be increasingly flaky and unreliable.

Leo Dion (host): Okay. That makes sense.

Rachel Brindle (guest): Yeah, disappointing. ~But it makes sense.~

Leo Dion (host): Yeah. ~Uh, ~before we close out, a lot of people are curious about like CUI tests. They're still very much tied to the old Objective C APIs and things like that. Where, what do you see as like the future of that as far as being more Swift friendly,

Rachel Brindle (guest): I mean, it's gotta be like Swift Testing support. There's a lot of, it is a lot of those APIs are very extremely typed. So ideally there'll be some way to make them. Not be as much there is. I don't really see how, I guess there is the, ~uh, ~what is it, the dynamic callable or like the dynamic property accessor from a while ago that could be used to at least make it feel more Swifty, even [00:36:00] if it doesn't actually provide any guarantees. ~That might be it. But. Or ~that might be it in terms of like making it feel more like idiomatic Swift. But like I think what they're more interested in doing is enabling, like using the automation framework with Swift Testing.

Leo Dion (host): Okay.

Rachel Brindle (guest): Because one of the things that they did, I don't remember whether it was 16 three or 16 four, but they pulled out or Xcode 16, three or four they pulled out the, ~uh, ~audit.

They made a new XC automation framework, which is separate from XC test, which is another one of those reading between the lines of like, they're clearly going in the direction of enabling automation tests with Swift Testing. ~Yeah.~

Leo Dion (host): What is the automation framework exactly?

Rachel Brindle (guest): It's all of the XCUI test ~AP ~APIs. So ~it's like getting, um, ~it's like the XCUI element, so you can get ~like ~an ~some view ~element by the, ~uh, ~accessibility identifier. ~Uh, ~it's like the XCUI application and ~like ~interacting with it through there. ~Yeah.~

Leo Dion (host): Is that used outside of [00:37:00] testing ~for like. Um, ~for like Mac OS apps.

Rachel Brindle (guest): ~I don't believe so. ~No, it's not. It's entirely meant for use with automation tests.

Leo Dion (host): Because I know like there's apps that I'll always ask for, like accessibility, access. ~Is that so that's something different.~

Rachel Brindle (guest): That is a different, that is also it's a different automation library from Apple. Great. ~The, ~so that is much more of like just being able to interact with the system through the. Automation A PII believe. I wouldn't be surprised if XE automation is fundamentally using like those same APIs under the hood,

Leo Dion (host): Under the hood. ~Yeah, that's what I was gonna ask. Okay. Okay. So~

Rachel Brindle (guest): but I don't have any way to verify that.

Leo Dion (host): ~but it's probably true. ~Anything else you wanted to talk about, especially with dub or testing before we close out?

Rachel Brindle (guest): Yeah, ~the, I guess the other stuff, you've talked about it with other folks as well. ~The other stuff I'm really excited for all of the, ~like ~concurrency improvements with Swift six two. I've been ~like ~diving into those and just been really excited for that. The defaulting to running everything or constraining everything to the main actor.

As opposed to ~like, ~non [00:38:00] isolation is something that I'm excited to see. Though it is right now very buggy. But you know, it's also just happened. It's early in the beta season. We'll see how it turns out.

Leo Dion (host): Well Rachel, thank you so much for coming on the show. ~Uh, ~it's great to have you on and talk about this stuff. Where can people find you online?

Rachel Brindle (guest): yeah, I am. ~Um, ~my blog is at rachel brindle.com. I am on, ~uh, ~Mastodon at unata@hackderm.io. And I. Instagram, though, that's mostly flying stuff now. ~Uh, ~at Rachel Brindle. Yeah.

Leo Dion (host): Thank you again. ~Uh, ~we'll put links to the proposal, your blog post, all that stuff in the show notes below. People can find me on ~Twitter at, oh god, Twitter, that's an old one. Whatever, ~Mastodon at Leo G. Dion at c Im~ uh, ~I'm on Blue Sky as well at Leo g Dion. Take a look at me there. Links are in the show notes.

If you really enjoyed this episode, please like and [00:39:00] subscribe. Early access is to those who joined Patreon, so thank you for all the Patreon supporters. It's super helpful. Thank you so much and I look forward to talking to you in the next episode. Bye everybody.