Leo Dion (00:00)
Hey, welcome to another episode of EmpowerApps. I'm your host, Leo Dion. I'm joined once again by the wonderful Daniel Steinberg. Daniel, it's good to see you again.

Daniel (00:11)
It's great to be back. Thanks for having me.

Leo Dion (00:13)
I'm super excited to talk about our topic today 'cause I think there's a lot of great value in it. and I know a lot of people are taking advantage of it. but of course we have to do the intro. So for those who don't know who Dim Sum Thinking is, I'll let you go ahead and introduce yourself.

Daniel (00:30)
Wow. so I've been around the Mac for a long time and I write books and teach and I contract and I just have a ball with this community.

Leo Dion (00:40)
Awesome. So a lot has changed since the last time you were on. I don't know if you've heard I probably use this joke too many times now, but like there's this AI thing now. where where do you stand when it comes to AI and where it's useful and where it's not useful?

Daniel (00:59)
So I used to teach classes with Matt Drance, and Matt had my favorite description of new technology. And it was stage one is new technology comes around and you go, my God, I hate this. I'm never going to use this. And then stage two, it solves a problem for you, and you go, my God, I love this. I'm going to use it everywhere. And then you come to your senses, and stage three is I'll use it where appropriate. And I'm trying to be in stage three.

But people who are in stage two think I'm in stage one and I just hate it. So I try to use it when it's appropriate. So for example, recently I wanted to change a Docker image so I could use it in Apple's containers for Jekyll. And so I asked Claude, hey, write me this thing. And it wrote me this thing, and then it wrote me more, and then it wrote me more, and it was awful. And then I realized all I really need

really needed to do was build the gem file lock for me and then I could do the rest. So I told this to a friend and he said, you're getting to the the right point where don't ask it to do the high level do everything for me. Figure out that really specific task you need from it and ask it to do that. And this way I didn't have to install Ruby and a bunch of gems to get the thing that I wanted. So I guess that's where I am right now where I'm trying to find out where's that sweet spot for me.

But I don't want it to write code for me in general. And I don't want it to write prose for me because for me, that's where I learn. When I struggle with things, that's how I learn and I get better. And I don't have a gig where I have to generate thousands of lines of code a day. I'm trying to write things and learn from and write things that I'm proud of. Does that make sense?

Leo Dion (02:47)
Mm. Yeah. So would you say you use it every day for small little things still? Okay.

Daniel (02:51)
No. No.

I use it as little as possible, but when I

Leo Dion (02:55)
Okay.

Daniel (02:56)
need it, I reach for it. So I I have it turned off in Xcode because I found A it was getting in my way and before this year it was suggesting APIs that didn't even exist. So at least it's not doing that anymore.

Leo Dion (03:10)
Yeah. Did you ever get that like when I do watch or Mac development, it'll just it used to do this, it doesn't do it anymore, but it'll hallucinate an API for watch or Mac because it thinks. It sees it for iOS and it figures out we'll just change the next or watch kit prefix at the end and there you go.

Daniel (03:28)
Or I'll type

in a method and it'll try to fill in the parameters and there's there's no API with those parameters.

Leo Dion (03:35)
Yeah, yeah, exactly. so now that we've gotten that out of the way, how how we're legally obligated to ask that question. How has has foundation models changed anything in your perspective on where it's useful and where it's not?

Daniel (03:53)
Yeah, so what I said was I don't want to use AI to write and generate code or writing or things I have to think about. But that doesn't mean it's not the right tool to use, say, to summarize stuff. I have a friend who's a speaker who will have one of the AI guys summarize what his talks about, and if their summary doesn't match what he thinks he's talking about, he needs to adjust. So it's a great tool for that. It's a great tool for going in in and finding

excuse me, finding a resource. so, you know, you you're researching something or something that's Siri AI, the new Siri I think is so much better than the old Siri. So I can ask it,

Leo Dion (04:31)
Agreed.

Daniel (04:32)
hey, can you ask Apple Music to suggest an artist like Miles Davis or in the vein of Miles Davis? So there are great uses for it and incorporating that into apps is is really nice.

Leo Dion (04:48)
So total aside question. Did you try or have you tried the thing in Apple Music where you're like, create me a playlist based on a prompt?

Daniel (04:57)
So I'm about to experiment with that because I wanna update so I have this free music app, so I'm not advertising anything because I don't charge for anything, called Top Tracks, where we build top forty radio stations out of Apple playlists. And so I wanna try

Leo Dion (05:09)
Okay.

Daniel (05:10)
that so I can say, Hey, can you suggest a playlist based on the things I tend to listen to or this this other playlist?

Leo Dion (05:17)
Yeah, w when I tried it a few months ago was it was not very good. So hopefully they've improved on that. I need to try it again. so yeah, let's let's get into it. Let's start off by just asking what what should developers know before getting started, especially those who've used these like AI tools. Like what what should they know about how foundation models work and what they can get out of it? I think that's right.

Daniel (05:44)
So first they should

just they should play. And one of the really nice things is #Playground makes that really easy. So #Playground is something that you can have something that fails inside of it or is async inside of it. So it's a great tool for just popping a page and testing your prompt, your instructions, and seeing what kind of things you get back. Also, I'm not as big on the command line tools, but you can use FM from the command line and experiment that way.

So the first thing is just to get in there and experiment, is it meeting the use cases that you want for your app?

Leo Dion (06:19)
Okay, so th is that just like a regular playground in Xcode I can play around with?

Daniel (06:25)
It's it's not. it it

Leo Dion (06:27)
Okay.

Daniel (06:28)
feels more like the Swift equivalent of preview. And so use preview

Leo Dion (06:32)
Okay.

Daniel (06:33)
and you experiment with SwiftUI code and you see it. So #Playground feels like the the Swift version of that.

Leo Dion (06:41)
Okay. Okay. I do I just do I but

Daniel (06:43)
What's nice is I believe I don't know if they're there

yet, because certainly Foundation Model isn't cross-platform, but I believe #Playground, they're working towards that being cross-platform. And so it's a great

Leo Dion (06:54)
Okay.

Daniel (06:55)
experiment. I I liked Xcode Playgrounds. I never liked the the more broad ones that you could use on iOS. I th I thought you know, I wanted more of a Jupyter notebook type of thing or or something more

Leo Dion (07:07)
Yep, yep.

Daniel (07:07)
full.

#Playground is a nice in between. You can put it in any dot Swift file and you can iterate. It's great for algorithms. It's that with with unit testing is just a a great tool.

Leo Dion (07:22)
So you'd have to like go into an actual project and like add a Swift file and then add the playground in there, essentially.

Daniel (07:28)
I don't know

if you can do it in a standalone dot swift file. I don't I haven't tried it, but perhap

Leo Dion (07:32)
Okay.

Daniel (07:33)
perhaps. I don't know.

Leo Dion (07:34)
Yeah, we'll see. So, you know, there was a lot of improvements in OS 27, over twenty six. what what did that get you that OS twenty six didn't get?

Daniel (07:51)
So our first iteration of foundation models was in 26. And I like that you're saying OS 26, because that's sort of iOS, macOS. It's it's it's all the OSs.

Leo Dion (08:01)
Right, right.

Daniel (08:03)
and there were some problems with it, but I'm okay with that. I I like that we're being exposed to it and we're seeing as it evolves. So some of the things are in OS 27, some of the things they introduced were so that.

The improvements don't have to wait a full year. Another thing that I think is a little confusing to people, it happened in OS twenty six four, I want to say, when they updated the model, that the model lives on the OS. So if you went to Xcode something, but your Mac wasn't updated, you weren't using compatible models and things failed. And so something you get you get your head around is not even in the simulator.

When you're doing the iOS simulator, it still depends on the version of it in the Mac. So that's that's right. So

Leo Dion (08:52)
In the host system. Yeah, okay.

Daniel (08:55)
one thing you have to get used to is is that that and I don't think that's very well documented, that that can trip you up easily. excuse me. Other things that we got in 27 was a better on-device model. We got the Private Cloud Compute, we got some tools that allow you to do interesting things with images.

We got the whole evaluations framework. we got some really nice APIs for building a more dynamic system. And then we got all the stuff that they're open sourcing that I would say is separate from foundation model and not what I want to talk about today, mainly because I don't understand it very well yet. but we we got an awful lot this year. It was a it was a big year for releases and so so what I do is is I tend to find what attracts me.

I teach it for a year, teach workshops for a year, and then I decide is it going to be a book? And this will be my next book. last year was just too early because you could see the holes that they had to fill. And I think they filled a lot of them at twenty seven.

Leo Dion (09:54)
Yeah, I we've been talking over the last year and I could tell like twenty seven was a big deal for for foundation models and yeah, it was probably the right decision to wait on the book regarding that. I don't wanna get into

Daniel (10:05)
Mm-hmm. I I still haven't

written my Swift 6 book because I don't consider Swift 6 complete yet.

Leo Dion (10:12)
I I agree with you. Yes. That's true. I don't wanna get

Daniel (10:16)
And I s and I and

and I actually should stop writing books because no one buys books anymore. They just buy tokens. So that's another thing.

Leo Dion (10:24)
Well,

you should sell tokens then.

Daniel (10:26)
That's right.

Leo Dion (10:28)
or sell skills, that's the other thing, right? So there's like a hierarchy too to what models are available on what device, right? So like what's it even if you can install twenty seven, that doesn't mean you're getting the best model. It depends on what what device you have, right? It's like correct correct me if I'm wrong.

Daniel (10:48)
So

I I don't know I actually don't know the answer to that question. What I do know is before you write any foundation model into your code, you should write some sort of a test into your app to make sure that the user can run stuff. And so

Leo Dion (11:04)
Okay.

Daniel (11:05)
you c you can query, you know, do you have a device that can run this? If you do, do you ha have you installed it? If you've installed it, has it downloaded and completely installed yet?

You know, until all those things have happened. And then there's the one that you don't test for but was a problem. I haven't tested this lately, but if your system language and your Siri language differ, it doesn't work. And so you've got to be careful about that. Even for a while I was

Leo Dion (11:32)
Who would what's the

use case for that?

Daniel (11:35)
Well,

Because I'm who I am, I had my system language as US English, but I was hoping when I drove round in my car that the car would say, You missed your fucking turn in Irish, so I had an Irish English instead. And it turned out no it doesn't swear at you just 'cause you have it set to Irish.

Leo Dion (11:55)
Mm I won't beep that by the way, so we're good.

Daniel (11:59)
A bo a boy can

d a boy can dream.

Leo Dion (12:02)
Ha ha.

yeah, 'cause I just remember like when they released it was like it was like this chart of like you have to have this Well, when did they first announce the Apple intelligence stuff? Was that like three years ago? Like twenty four?

Daniel (12:17)
So they didn't

they didn't give us the stuff for it, but in the meantime, a lot of the the devices that wouldn't support it, you know, we're not using any anyway. And if you look, like even the iPad mini, I think the new iPad mini they just announced comes with stuff that supports AI better. So they're building this neural engine stuff and and you know, more more

Leo Dion (12:39)
Right, right.

Daniel (12:40)
stuff into just even the base level devices. And so that's clearly the way Apple's moving. Also

If you if you updated to this to if you updated dubdub to the latest version of the OS, you saw how long it took for Siri AI to install and index everything. It took a long

Leo Dion (12:58)
Yes.

Daniel (12:59)
time. So it's a real rethinking of how Spotlight and and all these things work.

Leo Dion (13:06)
I still so I just updated today to 27.2 on my Mac and like I feel like every time I update Spotlight doesn't find the apps anymore and it has to like it's yeah, it's weird.

Daniel (13:17)
And I want

to say in my Irish voice, but I won't do it again. It's right effing there. You know, how can you not find it? I c

Leo Dion (13:22)
Right, right, exactly.

Daniel (13:24)
I can see it.

Leo Dion (13:26)
Right. Or it's in my history. Like you know I use Xcode every day. Like

Daniel (13:30)
That's right.

Leo Dion (13:31)
yeah. I wanna go back to the error stuff. How like how would you even handle that? Because I've I struggle with that myself where it's like I have this whole app built around foundation models. Like okay, what if they install it on a device that doesn't have foundation models? This is like my my virtualization app Bushel, right? I

stupidly forgot I didn't know this that you can target only ARM and now the app was out there and supported technically Intel, but virtualization needs ARM and like I basically have to say my app is useless on Intel. Like, what do I do what what would you suggest a app developer do, I guess, if they don't based on the different errors that

Daniel (14:11)
So you cer you certainly

Leo Dion (14:12)
they can get

No, no, I was just saying based on the different errors you get, like whether it's like the device doesn't support it, doesn't have it the download the model downloaded.

Daniel (14:21)
So if it's something that the

if it's something the user can fix, you should let the user know. If it's if it's you need to go to to enable this in your settings, you can let the user know. If it's an old device, maybe that too. Your own le Jeroen Leenarts, who runs do iOS, the conference coming

Leo Dion (14:38)
Mm-hmm.

Daniel (14:39)
up in Amsterdam, he just recently wrote a really interesting blog post about this where he said your app should be local first so that if foundation models isn't available.

You should have some things that work locally. And so it should default to nice, nice behavior. And perhaps you want to let the user know, but you don't want to let the user know every time. Once you've told them and they've decided, okay, I don't want to install this stuff, you still should should make some behavior available to them. The other thing I'll say is you know, we now have Private Cloud Compute, and you're tempted to always use Private Cloud Compute, but if someone's not on Wi-Fi.

That's not free. You know, they're using network traffic to download it. And if someone even is on Wi-Fi, there's a quota and they're using up their quota. So you should use Private Cloud Compute judiciously, not from your point of view. It's pretty much free to developers, but to the user's point of view. And you may want to alert them, hey, you're running out of your quota or you're running this over cell. Again, allow them to dismiss it, but let them know.

Leo Dion (15:48)
So it was like earlier this week. I remember that Gui had put out that two million kit. Do you know what I'm talking about? So he

Daniel (15:55)
No, I'll have to look.

Leo Dion (15:57)
had so I don't know if you remember Pl Private Cloud Compute had a limit of who could use it. It was

Daniel (16:02)
Yep. Yep.

Leo Dion (16:04)
like two million. And then Gui Rambo, who's been on the podcast and is pretty well known for hacking into stuff. Yeah, yeah.

Daniel (16:10)
Well he he's awesome. Yeah, he's awesome.

Leo Dion (16:13)
he had put out a Swift package to get around it.

but then I guess like Apple either they had removed the restrictions, correct me if I'm wrong if you know, they'd either remove the restriction or they like there was a it wasn't necessar because I went to the URL and I'm like it's not there and I asked he's like, Yeah, we don't I archived it because it's not really useful anymore based on the way things work. Is that is that still correct though that like Private Cloud Compute has certain restrictions as far as who could use it?

Daniel (16:43)
So the the two million restriction is interesting because that's an odd number. You have to have a pretty big user base to to be over two million. And so what it sounds like to me is if you're a big company with tons of users, you should be having your own system. I don't understand

Leo Dion (17:00)
No one hundred percent.

Daniel (17:02)
it though, because Private Cloud Compute is something that is metered from the user's point of view, not from the developer point of view.

So I don't know why it matters. Like it seems like that should be something that is available to everybody, but you know, it's not a technical issue.

Leo Dion (17:20)
Or it's just like and I'm sure this is w the way they would end up going eventually is like bill people if you have a app that's using it a lot, like turn it into a money making opportunity. Apple's really good at that. Like I'm surprised they didn't go that route where it's like, hey

Daniel (17:36)
But I really

I really like this other solution because you can put out an app and not realize how much users you're using it and all of a sudden you're being billed at a rate that you don't have income to sustain it and it and then you close down your app. And so we saw that with with maps and and other things that use queries that were metered or like weather apps. And so the the metering adds up and sometimes catches a developer by surprise.

Leo Dion (18:03)
But I I guess what I meant was like you meter you meter until you reach a certain you don't meter until you reach a certain threshold, right? Like CloudKit kind of. Where it's like so insignificant for most people, but if you're like Facebook then you think about options, you know what I mean?

Daniel (18:19)
And the quota is is odd, from the user standpoint because I believe it's based on requests, not on tokens.

Leo Dion (18:26)
Okay. That is weird. Yeah. so wh why why would you pick why would you pick Private Cloud Compute over just local models?

Daniel (18:38)
Well, for world knowledge, for example. So I have an example I I'm using in my workshops in a talk where I show the image of Van Gogh's sunflowers. And I ask locally, hey, you're an art historian, you're talking to a group of kids who are seeing this picture, describe it. And it says this is a picture of a of a sunflower, it's heavy use of yellows and thick brush strokes. Then you use Private Cloud Compute and you set the reasoning to deep.

And it said this is Van Gogh's sunflower painted in this year using this technique. And here's where it's exit. Like it has world knowledge, and all of a sudden you get a much

Leo Dion (19:15)
Mm-hmm.

Daniel (19:16)
deeper response. I do the same thing with an example where we're recommending wine and recommending it locally with less reasoning, you get very superficial. You want a red, you want a white. When we use deep reasoning on Private Cloud Compute, we get specific labels and years. And so

Leo Dion (19:34)
Okay.

Daniel (19:35)
you can get much better, but you've got to decide as a developer, do you care? I mean sometimes the context is such that you won't care if it's a deeper answer or not.

Leo Dion (19:45)
Okay, okay, got it. Can you should we get

Daniel (19:47)
Al also

it takes more time because not only is there a round trip, but if you said

Leo Dion (19:51)
Yeah.

Daniel (19:52)
not just the network, but if you said deeper reasoning, it's taking time. And you've seen

Leo Dion (19:56)
Okay.

Daniel (19:57)
this on Siri AI where it it says thinking and you think, No, you're not. Y you're off getting a cappuccino.

Leo Dion (20:05)
Yeah, it's as it's known to do.

Daniel (20:07)
Ha ha.

Leo Dion (20:09)
I've actually had the issue where like I've queried it and then it keeps going and going and going and I'm like I don't really care that much that I need you to like get rid of it and then Spotlight doesn't work and then like now I can't even launch an an I hate that. We'll see if they fix that in 27.2.

Daniel (20:26)
So one of the things that got one

of the things that got turned off during the beta cycle was for the terminal, you used to be able to use Private Cloud Compute and they turn that off. But if you set it and do a query, it just looks like it's thinking and thinking and thinking and it's not. It's not doing anything.

Leo Dion (20:45)
Okay.

Daniel (20:46)
You set it back to system and it gives you your answer like that.

Leo Dion (20:50)
So you're you're talking about like the FM command. So

Daniel (20:52)
Mm-hmm. Mm-hmm.

Leo Dion (20:54)
FM you still can't use Private Cloud Compute?

Daniel (20:57)
I believe the last time I tested it you still can't. I th I th they turned it

Leo Dion (21:00)
Okay.

Daniel (21:01)
off in one of the betas and I don't think they turned it back on. And

Leo Dion (21:04)
But you still

have #Playground though, right?

Daniel (21:06)
You still have #Playground. So from

Leo Dion (21:08)
Okay.

Daniel (21:09)
a developer's point of view, you can do that.

Leo Dion (21:11)
Okay. I wanna we should probably get into it, because to kind of break things down, but let's talk about like the different components and foundation models and how they work.

Daniel (21:21)
Sure. I love

the APIs. I think they've done a good job. And I think this year they made them even nicer. And so the

Leo Dion (21:30)
Yeah.

Daniel (21:31)
first thing you would do is you would you would create a session. And at the very base level, you provide the session with a prompt. And then that can fail. That can take a while. And then when it responds, you reach inside, get the content, and you can see the answer to the prompt or the question that you made.

That's the very simplest love.

Leo Dion (21:54)
So the one thing I thought was amazing was this whole Generable macro macro, right? where you can like say I want my output to be in this format, essentially.

Daniel (22:08)
So that's that's much further down the line because it it actually does three things really well.

Leo Dion (22:14)
Okay.

Daniel (22:16)
we can go there right away or I can sort of get you there.

Leo Dion (22:18)
Yeah, go there right

away. Yeah, yeah, yeah.

Daniel (22:21)
So one of the things is if you'd like, instead of waiting till it's thought all the way through, because you know the way these things work is it it gets the next few tokens, the next few tokens, and so it's gonna wait till it's all done and then give it to you. Or you can stream the the response. And so if you stream the response, you'll use this macro and it'll give you the partially generated along the way. Or you say, look.

I want you to reach inside this string of unformatted stuff and I want you to turn it into a Swift type if you can. And so you can say, Hey, here's what my Swift type looks like that I'm trying to get you to match, and it will give you that. And now you can use it just as a regular Swift type. It's sort of like Codable on on steroids.

Leo Dion (23:10)
Right, yeah, one hundred pr I like that. It is Codable on steroids.

Daniel (23:13)
The third place is really nice is if you're gonna use a tool. And so if you're gonna use a tool, you wanna take that prompt, which is usually an un specially formatted string, and you wanna turn that prompt into a type and give that type to the you can use the Generable macro on the way in. And so you can use it on the way in, you can use it on the way out, and you can use it for streaming, and it's

It's really one of the coolest parts and it makes so much of this nicer.

Leo Dion (23:42)
I guess like so you have the prompt, the prompt gives you a response. This is all done in a session, correct me where I'm wrong. And then

Daniel (23:50)
And so I think

of the prompt as belonging to the user. And so the

Leo Dion (23:55)
Okay.

Daniel (23:55)
other thing that we can do is we can set instructions. And instructions are written by the developer. So in my art example, I said, I gave you a picture to describe, but my instructions were, you are this type of person speaking to this type of of audience. I want you to do this. And so the instructions are supposed to always

trump the the prompt. And so if they're in conflict, the instruction should win, and this should keep the the user from being able to take you off on a on a really big tangent. So we

Leo Dion (24:29)
Okay.

Daniel (24:29)
have instructions. Instructions belong to the session. Prompt belongs to the response or the stream that you're trying to generate. And then tools also belong to the session. And then you can also have generating options.

And so you can say, look, I want the reply to only take this many tokens at the maximum. I want it to have this sort of temperature. I want it to, since everything's probabilistic, I want you to treat the probabilities in this way. And so you can tune the type of response you get using generating options as well. And then something they introduced this year with these excuse me dynamic profiles where you can combine these things and say, you know, here's what I want you to use in this situation.

And so you can change that along the way. So that's sort of a a quick summary of the things that that the framework allows.

Was that helpful?

Leo Dion (25:26)
It's very helpful actually. And one thing I want to say, you said the API is really good. I also like that they used terms that like people like all these terms still make sense if you're using Claude or OpenAI or anything. They

Daniel (25:39)
Yeah, they use terms of art.

Leo Dion (25:41)
right. Instead of just being like, you know how they are with their branding, right? It's like, let's come up with a nice, cool, cool term that that will market well.

no, these are all terms that we use throughout other AI.

Daniel (25:53)
But at the same time, some of the things

that they they did this year, so a l a lot of what they gave us are structs, a lot of what they gave us look an awful lot like SwiftUI now. So for instance, excuse me, we think of a prompt as being a string, but a prompt is actually a type, and you can put anything then in that is prompt representable, which it sounds very familiar to you. And so things that can be used as prompts.

might include an image. And now I can build it up the way I would build up a SwiftUI view. My prompt can have a I can use a dynamic prompt and build it up using a body just like I would build up like a V stack containing different things. I can now have this this thing that contains different things for a prompt do the same thing for instructions. So they've really been nice about giving us APIs that that feel very familiar to us as Swift developers.

Leo Dion (26:50)
I I think you know this, but I'm obsessed with result builders. So like yeah, seeing that it's like

Daniel (26:54)
And and so they're you they're

they're using result builders everywhere.

Leo Dion (26:58)
Yeah, I think that's a great use case for that, absolutely.

Daniel (27:00)
There's there's

a prompt builder, there's an instruction builder, they've they've really done a nice job. I've I've loved result builders since they were called function builders. It's just such a nice tool.

Leo Dion (27:11)
Or D SLs, right? Is that is that kind of the same thing? Kind of.

Daniel (27:15)
It it is. Th the idea is is you're if you've ever built one, you're basically taking this thing that can take multiple inputs, and in the old days, because no one wanted to write more than ten, you only got up to ten. But then we got some improvements in Swift 6, could have been Swift five, where we got generics that could be very variadic and all of a sudden we could have them take as many as we want. Like if you remember SwiftUI in the old days, a V stack could contain at most ten things because

Leo Dion (27:43)
Right. Because the generic

Daniel (27:44)
That's all we could do.

Leo Dion (27:46)
was capped at that, right? so I wanna take a step back again and what let's let's talk about the prompt itself. So the prompt is gonna be built based on what the user is just whatever you want the user I don't know, I don't even know how to put it. But the user is gonna have some input. That input is gonna be used to create the prompt, right?

Daniel (28:07)
The user can have input. You could do it you could control it on your end with a drop down list or build it up anyway. But the idea is sure. Sure.

Leo Dion (28:15)
Or even just like images like you said, or photos, right?

And then you basically create the prompt based on that, more or less.

Daniel (28:22)
That's right. That's right.

Leo Dion (28:24)
Okay. Okay. I'm getting this now. And then what would be an example though, I guess, of a tool for taking that input and formatting it so it could be used by the the model?

Daniel (28:39)
So the the hello world tool that they always demonstrate is a weather tool because you know that's really easy. another one that I often do is a map tool because I can query the map APIs. Well, Apple this year, as part of OS 27, gave us three tools. two of them came from Vision. One is a barcode tool reader, and one is an OCR reader. And so I can point I can point foundation models at an image.

And if it there's not a lot of text there, it's pretty good at reading it and figuring out what it is. If it's very dense or small text, the OCR reader is awesome. The barcode reader reads the barcode and then you use your APIs to query. So what product is this for? You have to have a database of that. It doesn't know that, but it can read the barcode or the QR code and and get you that information. And that's that's pretty cool. And then the final tool I think they added this year was a spotlight tool.

And so those three are built in. The first two you need to import Vision to use. And I always love when Apple takes stuff they develop for one device and they make it available to other devices. It's just it's a natural. It's kind of nice.

Leo Dion (29:51)
Yeah. I and now that you say what tool is, like now I'm like, okay, now I get it, because when I use Claude code it'll start using grep or like awk or like all these bash commands in order to do what it needs to do as opposed to like having the AI just burn tokens trying to look for text.

Daniel (30:10)
And

this is where this ties into if you do App Intents, you can expose things to Siri so that it you you basically can create things that you use tools for. So what a tool does, if you implement a tool on your own, you say, okay, here's the type of object I'm gonna pass to it. So you create a Generable struct, and then you create a function called call, and call takes the arguments, and now say you say, I wanna.

Enjoy gelato in Bologna, where's a good place? So you create a struct that has a city and the type of thing that you wanna look for. And now you can make a map query from that. So your tool is using what you can do in map kit and you're sort of the interface to it with this call function.

Leo Dion (30:59)
It's awesome. Yeah. So it's basically a way to do something that doesn't involve thinking but and avoid burning tokens for no reason and give you access to like another API or MCP, I would assume, something like that.

Daniel (31:16)
Yeah, that that's I mean, there's a couple things about the way people use AI, which which you've sort of brought up a couple times now. And the burning tokens is there's sometimes that instead of AI, it's just an algorithm. There's no AI

Leo Dion (31:29)
Right. Right.

Daniel (31:30)
in it. And it feels like AI and it feels magic, but we're using something that's built in that we can do as an algorithm. And so it's magic, but it's a different sort of magic.

Leo Dion (31:42)
Right, exactly.

Daniel (31:44)
So so dynamic profiles is really powerful. So an example I've been using in classes lately is I want a wine recommendation. And you've got your busboy, you've got your waiter, you've got your sommelier, and that's three very different levels. And depending on who you're talking to, you're gonna get different recommendations, but also based on what meal you're having. So you might ask one of them, you know, what wine goes with fish or chicken or beef or whatever.

What a dynamic profile allows you to do is to take a profile for each one of those types, put it into this unified thing that reads and can switch off of them. And maybe the sommelier uses Private Cloud Compute if it's available. If it's not available, it can't, so we can test for that. maybe the other one uses a different type and the reasoning level is different. So in generation options, maybe one uses

a very literal temperature. So their temperature is zero. They'll give you a very literal answer. And their profile, their their instructions might be, hey, you're a bus boy. You don't know anything about wine. You don't even drink wine, but you can give basic red or white answers. And so the dynamic profile allows the user of your app to switch what they're using within the same session. So it still has access to the context of what the last person

Leo Dion (33:08)
okay.

Daniel (33:07)
So you could say you could call

over the sommelier and say, Hey, I like where where the waiter was going with that, but could you build on that and give me some specific options? So it still has that context, even though you've switched the profile.

Leo Dion (33:21)
Okay, the context part, that's the important part, is like the different profiles still have access to the same context. Okay, that makes total sense.

Daniel (33:28)
And that's another thing that

we got in OS 27 is it's much better of knowing how much you have left. And so you know when you need to summarize or do something to your transcript and start a new session based on what you were doing before. The tools are better. They're there's I'm sorry, there's a new

Leo Dion (33:43)
Do they have do they have like a

summarized do they have like a summarized method you can call?

Daniel (33:49)
It it's it's more that you're asking it to summarize and and and to give you something.

Leo Dion (33:54)
Okay, okay.

Daniel (33:56)
and and because you might want to be specific. You might want to say, you know, so when you when you start the next one, still base it on the instructions. The instructions haven't changed, but only use the last part of the conversation or only use these you you can be specific about what you want.

Leo Dion (34:14)
Can we get into like the transcript and how or context and how that works and what what control developers have?

Daniel (34:23)
you have access to the transcript.

I don't know in terms of control, you now have a much better Instruments so you can see things as as they occur. if you look at the transcript, I find it instructive because it includes the things that it's using and you can see the size grow. you can now ask the size of it in terms of you know, how many tokens have you used.

this

Leo Dion (34:48)
Okay.

Daniel (34:48)
was stuff that we were supposed to have before, but it now works much more reliably.

Leo Dion (34:54)
W so streaming is a is a big deal. Like it's funny because you were saying you listened to Tim's episode that we did recently. That was a big part of Vapor 5.

and like server-side stuff. And we kind of got into a whole tangent about MCP and how that's a big driver of that. Where why do you think how is streaming useful if you're using foundation models?

Daniel (35:17)
So users can see the answer coming in in real time. And so a user could you could either set it up so they can interrupt it and and say, Well, I want to see this. So you ever use one of these these coding assistants and it's it's taking a long time and you think of something else you should have told it and you want to add to that. So so that's that's something you're able to do. the other thing with streaming is

Just like a website loading, the user might lose interest. This isn't what they want to do at all, and they want to navigate away. And so streaming doesn't hold them up until the whole answer is available. It's a better user experience.

Leo Dion (35:58)
do you you have control about whether it's becomes streamed or whether it becomes one okay.

Daniel (36:03)
Yeah. Your your your call

your call to the session is you're either responding to or stream response to. And if you stream, you're making a call, which you then consume. The result is an async sequence, which again

Leo Dion (36:17)
Yeah, I love that.

Daniel (36:18)
I love that they like when they introduced async sequence, it looked really limited, but it's showing up in so many Apple APIs. It's really nice.

Leo Dion (36:26)
Right,

right. Yeah, 100%. okay, I remembered what I was gonna ask. Does the transcript show you what it's thinking?

Can you Okay.

Daniel (36:37)
Don't know. I don't know. I'm sorry.

Leo Dion (36:40)
Yeah, I forgive you. It's fine. I just don't like like that's you're talking about being able to see what it's doing. Like if you can see what it's thinking, that might also be kind of helpful to like

Daniel (36:51)
Don't

I don't know that would be helpful. for instance when I'm using

Leo Dion (36:53)
Okay.

Daniel (36:54)
a coding assistant, it's nice for it to say here's what I'm trying.

Leo Dion (36:58)
Right, right. Yeah.

Daniel (37:00)
And

and and it's also horrifying, like, I don't think I gave you access to that.

Leo Dion (37:05)
Yeah,

right. No, don't don't hack into hugging face, please.

Daniel (37:09)
Ha

ha.

Leo Dion (37:10)
Yeah, so one thing I d I I love about your presentation you did at iOSDevUK was the whole stuff about Van Gogh and like images and stuff.

there's a lot you can do now with images. and I like that you talked about the whole tool thing. what I'll let you go ahead and expand on that. Like you so it could be part of the prompt, it could be part of the result. What else?

Daniel (37:32)
And so the result is one of the things that's new. the way they did images in the past, now they're you using Image Playground to do a lot of that. And so you can either have it look at an image and then take your prompt and the prompt in that image, which you put in as an attachment, is is what generates the image, or you could say, Hey, here's the image.

I want you to describe it or extract something. And now I want you to create a new image based on this description, give very different results. And you can ask for it like the Image Playground in different styles. And so they've sort of gone all in on on Image Playground. And so on on iOS, you just display a sheet and say, Hey, go go go think about this and and show me the result here.

Leo Dion (38:19)
Is it how is Image Playground? Is it like pretty good as far as generating stuff?

Daniel (38:25)
Fortunately it's not gonna put anybody out of business anytime soon, but

Leo Dion (38:28)
Right, right.

Daniel (38:30)
but it's it's fun to play with and you can say, you know, create a watercolor of this or a cartoon image of this and you know,

Leo Dion (38:37)
Okay, okay.

Daniel (38:38)
i i for now it's a toy, I think it will become better. But I love that they're giving us early access to things like this.

Leo Dion (38:46)
So w I I guess my c my question was and then Image Playground that's all local, right? It doesn't need Private Cloud Compute if you want to generate an image.

Daniel (38:58)
I don't know the answer to that either. I haven't tried I've

Leo Dion (38:59)
I mean I assume it's

mostly local, but like I'm I'm just curious.

Daniel (39:02)
But I don't know if if

if I've got no network access, I don't know if it works. I don't I've never tried it. It's the problem,

Leo Dion (39:08)
Okay. Yeah, okay.

Daniel (39:10)
it's it's in it's kind of fun. It's the problem that we're always connected. in the in the early days

Leo Dion (39:15)
Right, that's exactly what I was gonna say. Yeah.

Daniel (39:17)
of the web, I lived in Cleveland, I would go out to O'Reilly and I'd fly into San Francisco and like nobody there was using dial-up modems, and so they didn't understand what it meant to make a website faster. You know, so

Leo Dion (39:28)
Right.

Daniel (39:30)
So you're right, these are things I'll I'll have to test.

Leo Dion (39:33)
Flip on the I mean what do you do? Airplane mode, I guess, to test it? Yeah. Yeah, exactly.

Daniel (39:37)
Airplane mode

or you know, like we used to do the network link conditioner, which was always a great tool.

Leo Dion (39:41)
right, right, right. Yeah, yeah, yeah.

Daniel (39:44)
You know, what does this look like on on on edge or something?

Leo Dion (39:47)
speaking of which, I wanna hop back and talk a little bit more about testing.

Daniel (39:54)
so see for testing, the Instruments has gotten much better, but I don't think of Instruments as a as a testing framework. The problem with testing Apple Intelligence is it may not give the same answer every time you ask a prompt with a set of instructions. And so you can't test it the way you would do a Swift test, which says, I give you this input, I expect this output. And so you have to use a different style.

And it actually is a way of also making your instructions and your prompts better. It's this new framework called evaluations. And again, they've used a term that's that's common in this in this field, which is nice. and and they've also used the term hill climbing, which is as you see what kind of responses you're getting to these these prompts and these instructions, you can

tailor your instructions to be better and get better results. And so that's the hill climbing part. And so the test is you're testing something that's probabilistic. So you're, you're, you can't just say, well, it passed, I'm done. Test it again. Test it many times, test it many different ways because it's going to come out different. And so your whole strategy is different.

Leo Dion (41:09)
Yeah, I love that. Would you I I think I asked you this, but I'll ask you on the podcast. You wouldn't run this along with your typical unit test because of the probabilistic nature of it, correct? But you would run it maybe.

Daniel (41:22)
I wouldn't also

be also because it takes so much longer, but but what drives the evaluations are Swift tests. And so you write your evaluations which set everything up and and have all the structure, and then you have a Swift test that calls that and then runs them. And so it's it's to me like if you've done testing a long time, the difference between unit tests and sort of end-to-end tests.

You're not going to run the end-to-end tests every time, but you're going to run them nightly or you're going to run them regularly. For me, I'm running the these evaluations as I'm developing the prompts and the instructions. Once I've got it once I've got my app in production, I don't know that I'm running them very much at all.

Leo Dion (42:07)
Yeah, that makes a lot of sense. And it's almost like it's almost like UI tests too, where it's like you don't want to run it every time, just run it every so often to make sure everything is working. what what do you mean by made up answers?

You said, What about made up answers? FM is not designed for world knowledge.

Daniel (42:30)
So I have an example before there was Private Cloud Compute, where you go into a bookstore and you ask the friendly bookseller, Hey, can you suggest three books, say, in the mystery genre? And because it doesn't have world world knowledge, sometimes it would just make up books. Like in mystery, it would say, This book is by John Detective or Amy Murderer, and you go, That's not a real book. so

The thing I talked about one thing I didn't like about AI. The other thing I don't like about AI is it never says I don't know. It

Leo Dion (43:02)
Don't know. Yeah.

Daniel (43:04)
never says I don't know. And so it knows that it doesn't know because it knows its level of probability of certainty. And so you I would love to be able to say, if you're less than this certain, just sit there. Just tell me you don't know.

Leo Dion (43:19)
Yeah, yeah, one hundred percent.

There's a vertical and horizontal question I have. is this available? Yeah, I know. Sound like

Daniel (43:25)
Do do I have to turn?

Leo Dion (43:28)
a business speak garbage.

Is it available on all platforms and OSs? What's like what's the range on that? And then as far as you can plug in different models. That's I guess what I meant by horizontal and vertical.

Daniel (43:46)
So the plugging in different models is an aspect of this that I said is is not something I've played a lot with, and it it's in my list of things, but it's it's part of foundation models, but it's also part of the other frameworks that they introduced this year. And so and those are are promised to be open sourced, which is kind of cool. as as for all platforms, it's still Apple platforms. you know, that the the Apple

Apple has actually been a very good open source citizen when it comes to Swift and some of this these technologies, but their job is still to sell you Macs and iPhones and watches.

Leo Dion (44:24)
Yeah, yeah, yeah.

Daniel (44:25)
I mean that's that's where their business comes from. So it could run on the server side on Apple Silicon. It can't yet run on the server side on Linux. Some of their other APIs beyond foundation models.

are acknowledging that as much as they've tried, a lot of people are still doing ML using Python. And you're just not gonna get those guys to change. You know, Chris Lattner went over to Google and demonstrated much better performance with this stuff using Swift. And it just didn't sway people. It's and it's very hard when you've got things like Jupyter notebooks and you've got, you know, all all the Python libraries, it's very hard to get people to to to move over.

I think they're doing great work and I think they're good citizen. And what you're seeing in some of the new APIs is you're able to integrate better with stuff done in Python and you're able

Leo Dion (45:20)
Yeah.

Daniel (45:21)
to use other models. So they they've got this sort of protocols that other languages and other agents can plug into. And so you're gonna see more of this, but that's sort of beyond what I'm playing with now.

Leo Dion (45:36)
you said it's available on all platforms, so like this would I could do this on the watch.

Daniel (45:40)
All Apple platforms.

Leo Dion (45:42)
Right. I could do this on the watch, I could do this on Apple Vision, I could do this on tvOS, or like that's what I'm curious about.

Daniel (45:50)
So again again,

the only things I've played with myself are macOS and iOS, which I guess is iPad OS as well. but I I'm I'm guessing, I don't know, but I'm guessing because of what they announced with the new watch 12 and and series support for that, that I'm guessing you could use this on the watch as well, but I've not done it.

Leo Dion (46:13)
Okay, okay. We'll we'll have to follow up on that 'cause I'm curious myself how far you can take it. Without killing the battery.

Daniel (46:19)
Boy, this is like an hour of I don't know, isn't it?

Leo Dion (46:23)
No no no hey that's what makes you better than AI. Cause if I had you on if I had you on as a

Daniel (46:29)
I I need a

I need a t shirt that says that. Better than AI.

Leo Dion (46:32)
Yeah, yeah that's right.

Daniel, anything else you want cover before I let you close out and plug yourself?

Daniel (46:40)
No, e has this

made any sense at all?

Leo Dion (46:43)
Yeah, it's been amazing. I guess no, one more question. Where do you where are you like this is amazing developers need to use this that they're missing out on? Where do you see opportunities for this to be used that folks aren't really thinking of?

Daniel (47:00)
So I think a lot of this, the answer is future, because I think we're gonna get very suspicious of AI that doesn't run on our device that we can't run privately. So I think Apple hasn't nailed this this message yet, it hasn't really hammered it home, but the fact that this stuff runs on your device, given Private Cloud Compute, given that you can call it to other things.

you can still do an awful lot of work on your device and so you can access private information without exposing it. Like I get freaked out with some unnamed systems that it knows an awful lot about me and it seems to be cavalier about how it shares it with other aspects of the business. I think Apple's been a very good citizen about privacy and that's why I think Apple intelligence

is something that we should look at and and really care about. What it doesn't allow are shops where they have to do cross platform apps and they have to do the same thing on different like Android and iPhone. That's that's something I can help you with. But shame shame on you, you shouldn't be doing Android anyway.

Leo Dion (48:15)
That's right. That's right. Don't ever do whatever it is, like what forty percent of the smartphone market or sixty percent of whatever it's at right now. well, it's funny you mentioned privacy being a concern, but I do think I do think the chickens are gonna come home to roost and I think cost will be a concern as well where local models will be

Daniel (48:36)
I think cost is gonna be a huge concern. Apple did that demo and everyone laughed because the amount of Mac power that it needed to do that on device demo and I and I totally understand that, but at some point the cost of tokens for big companies is gonna be higher than investing in that hardware and doing it locally. I just recently went to a a Java Enterprise conference and a lot of their talk was on device work and and

you know, controlling the cost of those tokens. And I I think you're right. I think this is gonna be a big issue. I don't think we're paying the price of tokens right now. I think we're getting discounts

Leo Dion (49:11)
No way.

Daniel (49:12)
that that aren't gonna sustain.

Leo Dion (49:14)
Right, one hundred percent. Daniel, thank you so much for coming on. what do you got going on as far as books, workshops, and talks?

Daniel (49:23)
Well, thank you for asking. so I'm finishing my App Intents book and then I'm starting my Foundation Models book. And so if people want to follow along in tech review, just email me and I'm happy to to let you follow along. I'll probably start that in in December sometime. Workshops if I'm invited. I love to give workshops. as things change, I'm looking at different style books and I'm trying to figure out what I want to be when I grow up. And so I'm considering writing

a book on calculus that I've wanted to write forever and a book on statistics and maybe some graphic novels or some that have more artwork, even though I draw like a two year old. So I'm having a ball. I'm learning new things all the time. I love talking to people like you. I love our community and I love that I get to keep going to conferences. So that's pretty much what I'm doing.

Leo Dion (50:14)
Daniel, thank you so much for coming on. it's been great. definitely check out Dim Sum Thinking and his material there. There's a lot of good stuff. yeah. thank you for joining me. if you like this episode, like and subscribe on YouTube or give me a review, please. share it with your friends, share it with your family. and also thanks to all the Patreon supporters as well. if you haven't caught

my newsletter, definitely take it some time and subscribe to my newsletter. Link will be in the show notes below. Thank you everybody and I look forward to talking to you later. Bye.