s02e06 Dr. Aaron Shaver – Autonomous AI

00:00:00.000 All right, and welcome back to the Big Cheese AI podcast. I'm going to be your host today, 00:00:04.800 Brandon Corbin, because Sean is tired. 00:00:06.880 Sean has just returned from Puerto Rico. So today I'm joined with the normal people. 00:00:16.080 We've got Sean Heis. We've got Jacob Wise. And then our guest, 00:00:19.600 this is the normal, I apologize, I break things all the time. And our guest today is Dr. Aaron Shaver, 00:00:26.080 who is the CEO of prompt privacy. And if you've ever listened to the show, you've heard me drop 00:00:30.240 the name prompt privacy multiple times. And that is because I've been working with prompt privacy 00:00:35.200 now for, I don't know, six, seven, eight months, something like that. So Dr. Aaron Shaver, welcome 00:00:40.640 to the Big Cheese Podcast. And so today we're going to be talking about autonomous AI. 00:00:46.000 And but first we need to get through some of our news topics. And I hope you all are prepared, 00:00:51.840 I know you are definitely prepared. I'm ready to go for our news 00:00:55.040 So we're gonna run through it first anything you guys want to jump into before you started. Yeah, I can jump in I was doing the 00:01:02.360 Microsoft I don't know that Valley 00:01:05.960 Volley yeah, that's their 00:01:09.020 So they didn't even like it was it was volley 3 right and like like they're not even trying to disseminate from from dolly 00:01:18.120 - Right, they're like, it's close. 00:01:20.400 Some Microsoft naming team again, they're back at it. 00:01:23.960 They're really good. 00:01:25.880 But yeah, it's a recording, takes your voice, 00:01:30.480 and then they can reproduce it with like two seconds 00:01:32.760 of recording and do a really, really good job. 00:01:35.200 - Wait, so they can only generate two seconds? 00:01:37.160 - No, no, of train off of two seconds. 00:01:38.800 - Training, thank you. 00:01:40.200 I got words today. 00:01:41.720 So they can record two seconds of your voice, 00:01:44.880 and then they can reproduce it accurately off of that. 00:01:49.200 And they haven't released it because it's too scary 00:01:51.200 or whatever. 00:01:52.040 - Are you kidding me? 00:01:52.880 - Yeah, don't answer your phone people. 00:01:54.960 - No, no, it brings up a very interesting point, right? 00:01:57.680 Is that you can't, so we've always talked about synthetic data 00:02:01.360 is going to be a problem, 00:02:02.560 like it's gonna be the kind of the new malware, right? 00:02:04.920 Where you could, hypothetically, 00:02:06.160 if I can, I've got plenty of recordings of Aaron here. 00:02:11.120 I could absolutely take it, train it, 00:02:13.680 and then send a message to some employee 00:02:16.000 and be like, hey, if you guys could go to Target 00:02:17.880 and buy me a couple gift cards 00:02:19.640 and send those to me because I'm stuck out 00:02:21.560 in Belize or wherever. 00:02:23.360 I mean, that's gonna be a problem. 00:02:24.760 - Yeah, I'm getting, you know, 00:02:26.040 there's been so many data leaks and you can tell 00:02:27.800 if you've data's been leaked 00:02:28.860 'cause you start getting random texts 00:02:30.160 and calls from numbers that may not have been had before 00:02:32.960 and they're really trying to get you 00:02:34.480 to confirm some of the information now. 00:02:37.000 - Right. 00:02:37.840 - Right, they're trying to, 00:02:38.680 hey, is it just making sure this is Sean, right? 00:02:40.560 Like do not respond to that by the way. 00:02:42.400 'Cause they have like your email and your right. 00:02:44.700 - Just for them, you know to do this. 00:02:46.100 - You know to get to the point. 00:02:47.640 - But the thing that does scare me is like, 00:02:49.740 is they're trying, maybe some of them I feel like 00:02:51.560 are trying to get you to pick up the phone 00:02:53.200 so they can get that recording. 00:02:55.080 - Oh, I think so, absolutely. 00:02:56.640 - And there's no way around it. 00:02:57.840 Like that they're talking about embedding like, 00:03:01.960 what the hell do you call it? 00:03:02.800 - Watermarks. 00:03:03.640 - Yeah, watermarks, but you remember audio. 00:03:04.880 Did anyone ever use audio jungle? 00:03:06.440 - Oh yeah. 00:03:07.280 (laughing) 00:03:08.120 - Dude, how's your music? 00:03:08.960 - That sounds really good. 00:03:09.800 - Audio jungle. 00:03:10.640 - One point, oh. 00:03:12.040 Every five minutes or that voice would come in and over. 00:03:15.720 - So like, if your boss asks you to buy a gift card 00:03:18.040 and then it says, "Audi-o, jungle." 00:03:20.280 - I did have a new one. 00:03:21.360 - No, so one of the clients that I was working with, 00:03:23.760 I did get a message from the CEO 00:03:26.440 and he requested that I go and I get some gift cards 00:03:29.560 and I'm like, "Well, what is this about?" 00:03:32.080 And so like, I've been trying to like, 00:03:33.680 and sure enough, they went through every single employee 00:03:36.000 and this was a startup, right? 00:03:38.000 So we're talking maybe seven or eight people, 00:03:39.960 but they were able to pinpoint 00:03:41.240 that Dan was a CEO and that Brandon worked there 00:03:46.040 or somehow or not. 00:03:47.040 - Right, well, but like the targeting of something 00:03:50.200 that's small was just like, man, I get almost as- 00:03:53.600 - Can we just make Microsoft to create this, 00:03:56.160 use this technology so that we can all be on a team's meeting 00:03:58.720 and just send our alter ego? 00:04:00.560 - No, I think that's underway. 00:04:02.280 - Yeah, we don't actually have to go to the meeting. 00:04:04.520 We just have somebody that's modeled based off of us. 00:04:06.920 - I was, dude, who did that. 00:04:08.240 - Really? 00:04:09.080 he set up, he basically did the recording of himself 00:04:13.240 in like the normal team thing. 00:04:15.320 And then he basically set it on a loop 00:04:16.960 and he kind of configured it. 00:04:18.020 So he could then do basically, 00:04:20.360 his objective was how long could I go 00:04:22.400 through our team's meetings without anybody knowing 00:04:24.560 that I was never actually there? 00:04:25.400 - But I'm talking about actually interacting 00:04:27.840 and participating. 00:04:29.200 - But I think he did. 00:04:30.040 I think he did have like different mixes in there. 00:04:32.480 Now, giving us keeping it close. 00:04:33.880 - Right now. 00:04:34.720 - Right, it definitely gets there. 00:04:36.240 - I mean, is there a world, a future world 00:04:38.000 where you kind of have like a alter ego, 00:04:41.240 sort of the eye self that's doing like the stuff 00:04:43.120 that you don't want to do. 00:04:43.960 - But if it's all of our AI representatives, 00:04:46.600 then they don't need a teams meeting to work together. 00:04:49.000 - That's true. 00:04:49.840 - That's true. 00:04:50.680 - But we will, but we will, we'll be like, 00:04:52.600 no, we don't even have the teams meeting. 00:04:54.200 We need to have the human element on it. 00:04:56.160 But I'm just having my avatar go talk to you. 00:04:57.960 - And this one being person. 00:04:58.800 - There's some like intern project manager 00:05:01.600 that's trying to figure out what the hell he's AI. 00:05:03.160 - He's like, wait, hold on, hold on. 00:05:06.240 - I feel like it's trained to be an asshole, 00:05:07.700 'cause you guys an asshole. 00:05:09.800 - I say you that new French company that released, 00:05:12.580 I can't remember the name of it. 00:05:14.080 - Yeah, so, and I threw that on here, 00:05:15.720 and that was, yeah, it's basically the new AI assistant 00:05:20.720 that beat OpenAI, that they have a new model 00:05:23.760 that they've released that is doing the zero latency. 00:05:27.640 And so while it's listening, it's also kind of talking-- 00:05:30.680 - And that's open source, right? 00:05:31.800 - It is open source now. 00:05:32.800 - They haven't released it. 00:05:33.880 - Oh, that's the whole-- 00:05:34.720 - Yeah, it's open source. 00:05:36.120 - I'm not what you can't have. 00:05:37.620 - It's not open source then. 00:05:39.120 (laughing) 00:05:39.960 But when I went and I started playing, 00:05:41.380 I ended up sending Aaron Holyo, 00:05:43.220 I was like, "I'm just glad in my ass off." 00:05:45.920 'Cause I'm like, "Hey, tell me about, 00:05:47.720 or recite the Pledge of Allegiance." 00:05:49.320 It's like, "I don't think I can do that right now." 00:05:51.040 And I'm like, "Oh, that's okay, just recite." 00:05:52.320 No, I don't think I can do that right now. 00:05:53.780 I mean, it was like this very bizarre-- 00:05:55.320 - I've never explained my first question since you're, 00:05:57.760 I believe I'm dying to talk to you, by the way, 00:05:59.180 on the podcast for a long time. 00:06:01.640 Is like, why do some AI have such issues 00:06:05.660 doing math and do it with hallucinations. 00:06:08.100 Like what's the deal with like that particular problem? 00:06:11.180 Is it either this similar problem 00:06:12.540 or is it totally different issues? 00:06:14.300 Like, like-- 00:06:15.380 - Well, to be clear, all of AI is a hallucination. 00:06:20.220 Sometimes that hallucination lines up 00:06:22.180 with your perceived reality and sometimes it doesn't. 00:06:25.540 But it's not, I mean, it's all predictive. 00:06:29.620 And so a good example is like you guys are all programmers. 00:06:34.620 programmers. Have a large language model try to tell you the Unix timestamp of a particular date. 00:06:41.100 And it will be within five hours to three months inaccurate of what the timestamp is. 00:06:48.460 And that's because there's not a great training data set out there for how to go from the English 00:06:55.180 language to a Unix timestamp. >> Try doing a JavaScript. 00:06:59.100 - It's just trying to predict what that might be. 00:07:04.100 It's not actually performing the mathematical calculation. 00:07:09.100 - Didn't you see that the farther back the date went, 00:07:12.300 the bigger the drift? 00:07:13.820 - Yeah, because the further back you go, 00:07:15.180 there's less training data. 00:07:16.300 If you ask it, right, if you ask it, 00:07:18.700 like what's March 13th, 1965? 00:07:21.940 - Which actually is before the original time. 00:07:24.220 - It is, but those are just negative numbers. 00:07:26.060 And so if you're in Python, that's an easy calculation. 00:07:30.540 If you're using a large language model for that, 00:07:33.100 it's not even gonna come close to what it would be. 00:07:35.580 First of all, it's gonna give you a positive number. 00:07:38.380 Because it doesn't understand that it's like 1/1/1970 00:07:43.220 is zero time. 00:07:46.300 So it's always gonna give you a positive number 00:07:48.420 because all the data that's available for training 00:07:51.500 are positive numbers. 00:07:52.780 And it doesn't actually understand the number. 00:07:55.260 And that's kind of the big difference right now 00:07:57.480 between LLMs and this autonomous AI, 00:08:00.520 where they're trying to be more like a human, 00:08:02.360 'cause humans have the ability to plan their problem solving. 00:08:07.200 They have the ability to utilize tools. 00:08:10.280 And like you're talking about an LLM, 00:08:11.880 like it's not gonna, when you ask it to go do, 00:08:15.080 convert the Unix timestamp, 00:08:16.160 it's not going to its timestamp calculator tool. 00:08:19.000 - But we're stopping it from that because like, 00:08:21.240 it should, yeah, I mean, 00:08:22.840 - No rules trying to use things like that, right? 00:08:24.520 - That's the whole concept, right? 00:08:25.680 Is that they're gonna start using those tools. 00:08:29.320 - Okay, to be clear though, it can't. 00:08:31.680 - It can't? 00:08:32.520 - In and of itself can't. 00:08:33.920 - In and of itself doesn't have the agency 00:08:37.040 or the ability to go access a set of tools. 00:08:39.680 We have to give it those tools 00:08:41.600 and give it the impetus to use them. 00:08:43.880 - Mm-hmm, right. 00:08:45.320 - But in practice, like isn't that the biggest, you know, 00:08:50.000 the advancement was LLMs, 00:08:51.400 but then there's also an assumption maybe 00:08:53.320 that it is using these tools or it is doing the stages. 00:08:55.960 - Well, yeah, it's like-- 00:08:56.800 - You know what I mean? 00:08:57.640 - When we talk about Chad GPT, right? 00:08:59.360 So you have Chad GPT the model or JPT-4, 00:09:02.720 oh, god fucking name, the model, 00:09:05.360 but then you have the way that you're interfacing with it, 00:09:07.680 right, and that interface that they've got so much 00:09:10.720 like code just sitting on top of before 00:09:13.000 we ever even hit the LLM. 00:09:15.080 You know, so there's just a lot like again, 00:09:16.720 Roaldog in an LLM, you're gonna have one result, 00:09:20.160 but then you have the thing on top, which is the tools, 00:09:23.160 which is the saying, hey, first I want you to go 00:09:25.280 and identify what tools should I hypothetically use 00:09:27.480 for this user's prompt, right? 00:09:29.840 And then, oh, hey, he wanted to calculate a timestamp. 00:09:32.800 I see that I've got a timestamp tool in my array of tools, 00:09:36.120 and so I can just send that to there, 00:09:37.520 get the results back and I can answer that, right? 00:09:39.680 But again, that's all of the things that we sit on top 00:09:41.840 of these large language models, but yeah. 00:09:43.520 Well, I think you called it a super advanced auto complete. 00:09:48.880 - Yeah, yeah, I mean, it's basically what these LLMs are, 00:09:52.840 right? They're just super advanced, auto-complete systems. 00:09:56.160 - Yeah, I mean, that's the basic transformer architecture 00:09:58.600 is that they're just predicting the next token 00:10:00.640 in the sequence and then trying to manage a certain type 00:10:03.520 of attention with that. 00:10:04.680 - Yeah, yeah. 00:10:06.240 Now we did see, so we had one of the clients that we were out 00:10:09.680 where we were doing the AI immersion. 00:10:12.000 And I think we did talk about this a little bit 00:10:13.400 on the podcast is that we did find that like, 00:10:16.280 Gemini 1.5 Pro was better at kind of like discerning data elements from these kind of prompts 00:10:23.400 versus chat GPT, right? 00:10:25.200 So nobody in that team ended up using chat GPT for the work that they wanted. 00:10:30.240 And the one company who was trying to basically parse out or the one team who was trying to 00:10:33.160 parse out like CSV data and trying to extract some meaning from it, they ended up just completely 00:10:38.200 blowing away chat GPT, replacing it with Gemini. 00:10:40.640 And we're like, Gemini worked great. 00:10:42.480 And that was the problem is that chat GPT sucked is what they literally they said. 00:10:46.480 That was the problem. 00:10:47.760 So we do like pretty broad emergence where we take just business users and give them access 00:10:52.840 to tools and challenge statements to go kind of solve their own problems. 00:10:57.440 And less than 10% of the time people pick chat GPT as the model that they use. 00:11:06.000 You can see really in kind of more of the human related departments, you know, if you're 00:11:14.080 talking about like sales or marketing or human resources where it's very reliant on copywriting, 00:11:24.200 they really love the, yeah, theropic models. 00:11:27.600 If it's, hey, we're like logistics, supply chain, IT security, anything that's more of 00:11:33.560 of these harder sciences. 00:11:35.320 They love the Google Gemini models. 00:11:37.280 And that really just is, 00:11:38.800 it's not a function of architecture, 00:11:40.320 it's a function of training data sets 00:11:42.520 that they've been provided. 00:11:44.080 And probably the guard rails and shadow prompting 00:11:47.120 that those vendors are injecting into the models. 00:11:51.120 - Right. 00:11:51.960 So like what models are you guys currently using 00:11:54.760 for like your day-to-day work? 00:11:56.480 - Cloud Summit. 00:11:57.600 - Are you, have you kind of migrated to Cloud versus-- 00:12:01.040 - Mostly 'cause I started paying for it, 00:12:03.040 but I still have a tendency to copy and paste to all three 00:12:08.040 of the major ones. 00:12:10.480 - No you do, okay. 00:12:11.320 - Just to see. 00:12:12.320 - Or are you finding one that you kind of prefer 00:12:15.200 over the others? 00:12:16.720 - If I'm like trying to start from scratch 00:12:18.240 in like code stuff, Claude. 00:12:20.440 - Okay. 00:12:21.280 - But chat GPD for most general purposes. 00:12:23.000 - Chat GPD, the 4.0 is just sodium chatty. 00:12:25.520 That's my other problem with it. 00:12:26.560 It just won't shut up. 00:12:27.920 But I'm fine with that. 00:12:28.760 I'm like, do stop outputting the entire code. 00:12:30.840 I just need you to output the one like time. 00:12:32.560 - I think FORO has been perfectly fine for me. 00:12:35.200 - Yeah, FORO, like, I got mad at it yesterday 00:12:37.680 'cause I was trying to do some Zoom integration stuff. 00:12:39.960 And I'm like, no, the scope is not there. 00:12:42.200 Quit telling me it's available. 00:12:44.040 And then finally, they were like, 00:12:46.200 here's the webpage I found that it is there actually. 00:12:48.960 And I was like, damn it. 00:12:50.560 They're right, you have to enable this other scope 00:12:52.720 for it to show up though, which is helpful, 00:12:54.400 but I'm gonna add to it. 00:12:55.400 - I find that the thing that continues to amaze me 00:12:58.400 with FORO and I know the other tools are just as good. 00:13:01.760 like in Thropic, Cloud, whatever, 00:13:04.920 is just the ability to write code that works 00:13:07.840 in languages I don't know. 00:13:09.160 Like we were always doing some Python work 00:13:11.240 'cause I got an idea and I just tested it out 00:13:13.840 and I had a working app in a couple of minutes. 00:13:18.840 - Have you noticed the degradation in Co-Pilot at all? 00:13:24.000 - I don't use it. 00:13:26.960 - Mine's. 00:13:27.800 - I find that my issue, I'm more like you 00:13:29.720 when it comes to the coding. 00:13:30.680 and I don't like it in the IDE, 00:13:32.360 'cause I haven't found a good interface for that to happen. 00:13:35.600 - Yeah, that helps. 00:13:36.440 - It just doesn't feel right. 00:13:37.600 So I'm continuing to utilize the chat prompt 00:13:41.760 and have a conversation with it 00:13:43.480 and then bring that stuff in and out. 00:13:45.400 - The only thing I really use Copilot for now 00:13:47.360 is if I have a terminal error that I can copy and paste 00:13:50.120 into the chat feature and be like, 00:13:51.720 okay, given my code and this error, where should I go first? 00:13:55.480 - And I've had some regressions too, 00:13:56.840 like I used to have a really good auto-complete-- 00:13:59.080 - It only sucks all of a sudden. 00:14:00.440 - Yeah, it's terrible now. 00:14:01.280 - I don't know. 00:14:02.100 - So that's a very concerning point here. 00:14:04.840 I think that's worthy of a discussion, 00:14:06.600 which is that no updates to the model, 00:14:09.680 again, we've all been using 4.0, 00:14:11.080 I assume that's been using 4.0 or whatever it is, 00:14:13.880 but then all of a sudden out of nowhere it seems, 00:14:17.080 that it just starts to fall apart, right? 00:14:19.760 Like it starts to degrade, it starts to not get, 00:14:22.480 now is that a scalability issue? 00:14:26.000 Like what actually causes it 00:14:28.560 to start to really degrade over time, 00:14:31.080 because again, as you and I have talked about a lot, 00:14:33.640 is that when we have an enterprise, right? 00:14:35.640 Enterprises are going, 00:14:37.000 when enterprises really start to leverage AI 00:14:40.200 in their operations, 00:14:41.520 they're going to have to be able to really report on these, 00:14:44.280 right? 00:14:45.120 They're gonna need to be able to say, 00:14:45.960 this is the model that we're using, 00:14:47.360 here's the prompt that we're using, 00:14:48.680 here's the data that we're using, 00:14:50.240 and that we can validate that all of this 00:14:52.120 is actually accurate. 00:14:53.400 But when all of a sudden these external models 00:14:55.680 are starting to kind of drift or fade 00:14:57.680 or whatever the hell is going on, 00:14:59.600 that seems like it could be a really big problem 00:15:01.880 for the entire market. 00:15:03.040 Do you have any thoughts on how that is happening? 00:15:05.680 - Yeah, I mean, especially as we move away 00:15:08.080 from like the whole zero shot chat style, 00:15:11.040 I mean, that's fine for maybe some more sophisticated folks 00:15:14.000 or if you're trying to just solve casual problems, 00:15:17.200 but you really wouldn't want somebody in a clinical 00:15:19.720 or a business setting, you know, 00:15:21.920 performing, you know, producing financial reports 00:15:25.680 out of a chat style LLM. 00:15:28.480 Because there's very discreet like business rules 00:15:30.880 that we have to follow. 00:15:32.520 And so when you build out that symbolic reasoning 00:15:35.760 about how to produce a P&L or how to validate two invoices, 00:15:40.760 it's kind of like playing clue 00:15:45.640 where it's like Colonel Mustard in the kitchen 00:15:48.360 with the canvas there, right? 00:15:50.760 You have to say it's Gemini 1.5 00:15:54.240 of a particular release train with this in-context learning 00:15:58.960 with that retrieval augmentation of data. 00:16:01.880 And that's to remain relatively static. 00:16:04.640 If you change any one of those components, 00:16:07.380 really that prompt or agent needs to go through revalidation 00:16:11.920 to make sure that it's producing results that are consistent 00:16:15.560 with the rules that are in place. 00:16:17.680 And so that's where for business purposes, 00:16:20.160 we prefer models that don't drift. 00:16:22.480 I'd rather not have streaming fine-tuning with the latest web scrape data because that's going to create that drift in those model weights. 00:16:32.860 I'd rather have it be static and then control the retrieval augmentation of the data so we can make sure. 00:16:38.560 In those cases, if you're trying to do something that you're going to put your company's financials on it, it would have to be static. 00:16:45.220 You would have to be able to have a control. 00:16:47.880 - Well, that's like with any tool. 00:16:50.440 I mean, if it was a hammer and you picked it up 00:16:52.160 and it was different, every-- 00:16:53.440 - Yeah, different way. 00:16:54.760 - Different way. 00:16:55.600 - Different way. 00:16:56.440 - It was a ball of pain versus a claw. 00:16:59.040 - I also think that the way that that's used 00:17:03.480 is also consistent with the way people 00:17:05.160 are currently using it, which is they're just trying 00:17:08.960 to get stuff generally done quicker. 00:17:11.640 - Right, yeah. 00:17:12.480 - We're talking about building an automating 00:17:15.240 like business processes. 00:17:17.280 - So healthcare is a big area. 00:17:19.040 - Autonomous, you know. 00:17:20.040 - So for whatever reason, they're all going after healthcare. 00:17:22.880 Right, like so IBM made an announcement 00:17:24.640 that they're going after healthcare, OpenAI made an announcement. 00:17:26.680 - Well why are they going after healthcare? 00:17:28.120 'Cause that's where all the money is. 00:17:29.240 - I know that's where all the money is, right? 00:17:31.440 - I don't know about that guys. 00:17:32.520 I've worked in healthcare a bunch, they're straight up broke. 00:17:35.680 - Oh yeah. 00:17:36.520 - Well, depends what side payer or provider. 00:17:38.720 (laughing) 00:17:40.240 - Right, but I mean, if you wanna talk about pharma, 00:17:42.160 that's where the money's at. 00:17:43.240 - Sure, sure. 00:17:44.760 But again, they're all kind of starting to set up 00:17:48.280 these operations to basically be able to do healthcare. 00:17:51.160 I mean, it seems like, for me personally, 00:17:54.200 I got of all the things that we could go 00:17:55.880 and try to pick to go, like maybe healthcare should be, 00:18:00.240 I don't know, third or fourth. 00:18:01.920 Like, let's get something that's not necessarily, 00:18:04.880 'cause again, we've seen the horror stories 00:18:06.400 of what happens when, like some, I can't remember, 00:18:08.560 it may have been like five episodes, five episodes. 00:18:10.840 - If you tell a health insurance company 00:18:12.960 they're gonna be able to pay 20% less on, 00:18:15.560 or have 20% less claims, or pay 20% less, 00:18:18.800 or if you go tell a drug company 00:18:21.160 that they're gonna be able to enhance their drug pipeline 00:18:24.680 by, you know, whatever, they're gonna give you money, right? 00:18:28.920 I mean, you're gonna, so that's where, 00:18:31.480 so the reason why I would see that, 00:18:32.880 but yeah, I mean, I would hesitate to say 00:18:35.800 that that might be the right move in terms of like, 00:18:38.400 you know, risk or society. 00:18:39.800 - One of the areas that I know that kind of came up 00:18:42.400 while we were kind of talking through some of the healthcare stuff, there's areas of being able 00:18:46.320 to predict like how fast people are coming in and out of the hospital. Like those types of things 00:18:51.440 seem to be ones that are a little bit less problematic if something gets awry. 00:18:58.240 - To your point, they're the ones that don't have the money to pay that. 00:19:00.240 - We got to be clear in healthcare though because we're treating it like it's one thing 00:19:06.000 and it's not. I mean because you have the business administration of healthcare, 00:19:10.720 You have clinical administration, which is what you're talking about, like discharge and reception. 00:19:17.120 But then you have actual clinical care delivery. And there are different risk 00:19:22.560 tolerances and different benefits in those subsegments of it. And so it's different to think about 00:19:29.120 how do we use AI in general finance practices and health care that's kind of the same as every 00:19:35.440 other business. Or maybe you're doing like claims reconciliation. That's more of a business process 00:19:42.000 that's fine-tuned for clinical aspects. But when you go into a clinical decision support system 00:19:48.720 for a physician to help like recommend drugs or treatments, that's a different risk tolerance. 00:19:54.640 But like it's like a risk reward kind of thing. So yeah. Yeah, risk being you could 00:20:03.440 - That's gonna accidentally kill someone. 00:20:05.600 Reward being you could save someone with an idea 00:20:08.320 that you might not have. 00:20:09.880 - Yeah, it depends on how you look at risk though. 00:20:11.760 I think the big risk in that area is 00:20:14.560 will physicians and caregivers actually accept that tool? 00:20:19.280 You could invest a hundred million dollars in a model 00:20:22.600 and then they just say, we're not interested. 00:20:25.760 Like what is the path to revenue on that? 00:20:28.160 - True and coming back to the difference between the LLM 00:20:32.160 and the autonomous agent trying to truly replace 00:20:35.320 somebody's job is, it's not just the information 00:20:39.160 that they read in the book, it's their experience 00:20:42.400 and their interpretation of their own experience 00:20:44.360 and those memories that they have 00:20:45.680 that influence their decisions. 00:20:47.040 - Yeah, but now you're getting into reasoning process. 00:20:50.160 - Right. - Yeah. 00:20:51.520 And if you've met one doctor's diagnostic reasoning, 00:20:54.760 you've met one doctor's diagnostic. 00:20:56.840 (laughing) 00:20:57.840 It's not like they all go to med school 00:20:59.640 and are like, okay, we're gonna think about problems 00:21:01.640 the same way. 00:21:03.200 And so, like when you move towards that level 00:21:06.880 of personalization of artificial intelligence, 00:21:09.800 the consumer needs to see it represent 00:21:12.000 their own belief system and values 00:21:13.800 and processes and decision making. 00:21:16.480 - Right. 00:21:17.320 Well, that's kind of interesting because a lot of people 00:21:18.880 have that calm when they go to the doctor 00:21:21.200 and they have a serious issue. 00:21:23.040 They might have a specific experience they've had 00:21:25.360 with cancer in their family and how it was treated. 00:21:26.760 - Absolutely. 00:21:27.600 - And they might want that to be implemented 00:21:29.600 in what they're doing in their community. 00:21:30.440 There's also like cultural sensitivity, right? 00:21:33.280 And religious sensitivities. 00:21:34.860 And when we have like these, 00:21:36.460 like one of the downfalls for me about just like these 00:21:40.260 overfit general purpose LLMs is, 00:21:43.600 that's based on the flawed assumption 00:21:45.500 that we're all gonna think about things the same way. 00:21:48.240 And that's like, that's never gonna happen, right? 00:21:53.440 And does that mean that we then like train an LLM 00:21:56.640 for every person? 00:21:58.520 Like that also doesn't seem like 00:22:00.240 - That's gonna happen. 00:22:01.320 - Why is my L.O. I'm so paranoid. 00:22:03.240 - So what you're saying is it's not hallucinating, 00:22:05.080 I just don't understand your stuff. 00:22:06.480 - It's just crazy. 00:22:07.840 - But if you overfit a model, 00:22:09.600 let's say we all just everyone in the world 00:22:11.680 agreed to use chat GPT. 00:22:15.480 And like, how do you reconcile the differences 00:22:18.720 between Christians and Muslims 00:22:20.280 or Democrats and Republicans? 00:22:22.920 Like these are existential problems, 00:22:25.160 we're not gonna reconcile with an LLM. 00:22:27.280 - Right. 00:22:28.120 - And so when we start to move closer 00:22:29.560 closer to those more humanistic applications, we have to look beyond just like the predictive 00:22:37.000 trait. 00:22:38.000 You can't get from big data to cognitive reasoning, right? 00:22:44.800 Our data doesn't represent our reasoning. 00:22:47.960 So if I gave you all of my financial transactions, you guys could relatively quickly fire up SageMaker, 00:22:54.880 train a predictive model on it. 00:22:57.040 But you can tell me, is why I made the decision to spend that money. 00:23:02.800 And so there's situations where we're all kind of under this shared delusion that if 00:23:08.320 we collect enough data, we can figure out how everyone makes their decisions. 00:23:13.880 Which means we cannot use AI in situations that have sparse data. 00:23:20.080 But that's what AGI is purported to be. 00:23:23.320 if it's gonna have the general intelligence of a human, 00:23:26.240 it has to be able to work in environments with sparse data, 00:23:30.240 which means it has to apply inductive 00:23:32.680 and deductive reasoning processes. 00:23:35.200 And LLMs don't do that. 00:23:37.120 - Yeah, go ahead. 00:23:38.560 - Well, I was just gonna say, 00:23:39.400 could they ever, 'cause a lot of LLMs are plausible, right? 00:23:43.280 Like we look at their output and we're like, okay, 00:23:45.000 it looks plausible. 00:23:46.800 I'm wondering if the future of AGI is really not real, 00:23:50.320 but it's plausible emotional reactions 00:23:53.520 and kind of how they reason about things, right? 00:23:57.880 Like you can inject chaos into it. 00:24:02.240 And then the output is like, well, I guess 00:24:03.880 that that person was a little bit unpredictable, 00:24:06.080 like we are, right? 00:24:07.040 - Yeah. 00:24:08.600 But that's where we get into neurosymbology 00:24:11.080 of how do we, like we have to teach our ais 00:24:13.880 to think about our environment and our world, 00:24:16.880 the way we think about our environment and world 00:24:19.680 because in order to represent our belief systems 00:24:22.240 and our viewpoints, it doesn't mean it's correct. 00:24:24.120 - Yeah. 00:24:24.960 - That means that's a barrier to user adoption that we have 00:24:27.600 is we have to feel like it represents us. 00:24:31.240 - Right. 00:24:32.080 In order for us to have those like truly, 00:24:34.960 like you said, reasoning type experiences 00:24:37.080 or situations where you're gonna give something, 00:24:39.600 a lot of responsibility, is it gonna have to really 00:24:42.960 like follow you along personally 00:24:44.760 or follow your business along for the ride 00:24:46.720 like in just like build up that knowledge with you? 00:24:49.560 Well, yeah, I think that's an interesting discussion 00:24:52.360 of the state of the industry is, 00:24:55.160 and I was telling Brandon this earlier, 00:24:56.760 we work with a huge global auto manufacturer. 00:25:01.760 And as we've gone through, 00:25:03.680 everyone's really concerned about their data, right? 00:25:06.800 Of like, is it secure and is it private? 00:25:08.800 And that's, like, if you're doing it right, 00:25:10.960 it's a bit of a non-issue. 00:25:13.200 But then as we're building out all of these agents, 00:25:16.400 one of their attorneys came back and they were like, 00:25:18.960 So based on what we've learned, we should be concerned 00:25:21.320 about the security of our cognitive reasoning, 00:25:24.160 like the way we think about things, 00:25:26.680 not as much can somebody get to our data. 00:25:29.600 Because that's really what embodies like 00:25:32.760 how we operate as humans and as organizations 00:25:36.240 and the culture and the success or failure 00:25:38.680 that's driven out of that. 00:25:40.040 And so we call it like cognitive architecture 00:25:42.920 or designer thinking. 00:25:44.460 In the past, you had to like go out and hire 00:25:48.560 The smartest people are the most intelligent 00:25:50.580 or the most well-educated. 00:25:52.200 But if I can custom design a cognitive reasoning process 00:25:56.440 and give that to you in an agent or a prompt, 00:25:59.600 you can use it at massive scale for low cost. 00:26:03.120 And that is the transformational power 00:26:06.020 of artificial intelligence. 00:26:07.540 But that means we have to catalog and design 00:26:10.440 and curate cognitive reasoning. 00:26:12.480 - Which hasn't been-- 00:26:13.800 - Which is what we work on. 00:26:15.000 (laughing) 00:26:16.000 - This is the path to AGI. 00:26:19.920 And I would argue that because you guys have worked 00:26:23.560 on what we would consider perception models. 00:26:26.240 So we don't get into the like, is it an LLM or is it an NLP? 00:26:30.840 Like there's so much religion around what you call it. 00:26:33.040 We just like broadly group them in like optical character 00:26:37.360 recognition and certain forms of NLP, 00:26:39.640 we would call those perception models. 00:26:42.480 And then when we get into different forms of LLM 00:26:44.960 transformer base, we would call those comprehension models. 00:26:49.000 And so these are different categories in different ways. 00:26:51.440 And if you look in the human brain, 00:26:52.880 this is how the human brain works as well. 00:26:55.920 Like if I gave you an OCR model, if you just trained it 00:26:58.440 and trained it, would it ever become a comprehension model? 00:27:02.720 No, like it has one job. 00:27:03.920 And its job is to be able to read characters and perceive them. 00:27:07.960 And it would never come to the point of comprehension. 00:27:11.560 And so there's higher layers of proto-cognition 00:27:14.920 that we have to look at. 00:27:16.280 And there is no amount of fitting or training 00:27:19.880 that you can give an LLM, 00:27:21.840 where it becomes a projection model 00:27:24.920 or a direction model or an action model. 00:27:27.760 These are different types of AI 00:27:29.320 that work together with language models. 00:27:31.520 - So really where we're at is broadly is that the LLM 00:27:38.920 came in and brought that different type of model 00:27:43.440 and that interaction. 00:27:44.280 In order for us to get to AGR, get to where we wanna go, 00:27:46.540 we have to build something even more sophisticated. 00:27:49.520 - There's, I won't even say it's more sophisticated, 00:27:52.080 there's just more work to be done. 00:27:53.640 - All right. 00:27:54.480 - So it's not gonna require some unbelievable advancement 00:27:59.480 and computing or thought. 00:28:01.680 - Well, so I'm kind of here to do. 00:28:02.960 - Do you think we get to an AGI? 00:28:04.960 - Yes, you do. 00:28:05.800 - Okay. - I do. 00:28:06.640 And do you have a prediction of when that might happen? 00:28:10.780 Roughly. 00:28:11.780 By which end? 00:28:14.780 10, 20. 00:28:15.780 And I buy. 00:28:16.780 You know, I don't think it's a matter of compute limitations. 00:28:23.060 I think we have the computing horsepower. 00:28:25.780 I think it's an issue of industry focus. 00:28:29.500 If you think that organizations that have been the most 00:28:31.860 successful in comprehension models, 00:28:34.220 their mission, if you look at Google's mission, 00:28:36.220 It's to gather and organize the world's data. 00:28:40.020 We have organizations that are now, 00:28:42.340 hey, it's our mission to gather and organize 00:28:44.660 the world's cognitive reasoning. 00:28:46.460 - We do. - And who are those companies? 00:28:48.980 - These are all like-- 00:28:50.860 - Startup. - Stealth startups, today. 00:28:53.140 - So people are out in that problem space. 00:28:55.340 - Absolutely. - Okay. 00:28:56.340 - Yep. 00:28:57.340 Essentially just like, we use web scraper 00:28:59.860 to like harvest your data. 00:29:01.740 We have new AI models and technology 00:29:04.020 that harvest your cognitive reasoning 00:29:05.940 of like how did you come to these conclusion 00:29:08.700 and how do we create the neurosymbology in code 00:29:11.940 and models to repeat that? 00:29:13.900 So when it makes a decision, 00:29:15.300 'cause you guys have all gotten results from a model 00:29:17.740 and are like, "Explain how you came to that conclusion." 00:29:20.100 And it just is like, "I'm sorry." 00:29:22.260 (laughing) 00:29:23.260 It like just apologizes that it was wrong. 00:29:25.740 - I'm sorry. 00:29:26.860 - Yeah, it's like, "I'm sorry, you're right." 00:29:28.380 That is the wrong answer. 00:29:29.580 I'm like, "I didn't say it was the wrong answer." 00:29:31.340 I just want you to explain how you came to that. 00:29:33.580 - It is so funny when you get a response from the prompt 00:29:38.580 and then you start chucking around 00:29:39.840 and coach it a little bit. 00:29:40.980 It almost always tries to immediately correct it. 00:29:43.820 - It's a total little bitch. 00:29:44.900 - It's like, yeah. 00:29:46.740 - It's like, wait, I didn't even really ask you to, 00:29:48.420 'cause you updated code that does, no, no, wait, 00:29:50.460 I was just saying. 00:29:51.300 - From a psychology perspective, 00:29:52.820 we're like, you've had a very traumatic ass. 00:29:55.220 - I love it, it's such a people pleaser, 00:29:57.420 do you have a trauma? 00:29:58.820 - I mean, how much of that comes down to the way 00:30:00.460 it was either trained with a reinforcement piece 00:30:03.540 Or was it really just kind of the layers on top 00:30:07.740 before you get to the-- 00:30:08.580 Yeah, so it's a combination of train. 00:30:10.580 It's a few things. 00:30:11.940 First of all, if you just look at the data, it's trained on. 00:30:15.620 It's all a huge portion of it's from the public internet. 00:30:19.220 And if you've met people on the internet, 00:30:21.340 they're kind of bitches. 00:30:23.780 So there's that. 00:30:25.220 That's inherent. 00:30:26.660 Then it's been fine-tuned with reinforcement learning 00:30:29.540 to like coach it to the zeitgeist of our era. 00:30:33.600 But then there's shadow prompting on top of all of it. 00:30:36.340 - Right, which is what we talk about a lot. 00:30:37.180 - You put all of that together 00:30:39.040 and we've kind of like neutered a bunch 00:30:41.420 of the capabilities. 00:30:42.420 - All right, right. 00:30:43.460 - Now, so we talk about, 00:30:46.660 and explain the concept of reasoning and all that, 00:30:49.380 but if we talk about, you know, 00:30:51.660 really autonomously getting things done, 00:30:54.300 the thing that's always interested me the most is, 00:30:58.020 is the tooling and the interaction with the real world. 00:31:01.980 And in order for me to get my job done, 00:31:05.460 I don't just get out a piece of shit, 00:31:08.780 I don't just talk, right? 00:31:10.260 I don't just get out a pen and a paper and write 00:31:12.740 and hand deliver it to the next person that needs a write. 00:31:16.300 I use a computer, I use application, 00:31:19.420 not really, I just go use chat, 00:31:23.780 GPT, I'm not gonna waste any new email, 00:31:25.780 that's my job, those days, no. 00:31:27.140 So like the concept of actually getting things done 00:31:32.140 is such a broad, you know, kind of aspect of AI. 00:31:37.740 And is it the easier thing? 00:31:39.700 Is it the more low hanging fruit of this AGI? 00:31:42.380 'Cause to me an AGI needs to be able 00:31:43.860 to get things done for me. 00:31:44.940 - Yeah. - You know what I mean? 00:31:46.340 - Yeah. 00:31:47.180 - And so I don't know is that we talk about, you know, 00:31:50.620 it's not necessarily going to a website. 00:31:52.700 It's gonna memorize your API. 00:31:54.900 And just in, is that more of a human problem? 00:31:58.220 Like we have to build systems that the AGI 00:32:02.100 can interact with from a tooling perspective, 00:32:04.660 or is it just gonna learn how to do stuff 00:32:06.460 and figure out, is it sophisticated enough 00:32:10.420 that it's just gonna be able to be like, 00:32:11.580 I'll figure this out. 00:32:13.260 - Well, probably a bit of both. 00:32:15.860 I mean, we do have, we invest really heavily 00:32:18.460 into what we consider like cognitive storage technologies 00:32:22.540 because we just don't believe in humans abilities 00:32:24.980 to find and organize their data. 00:32:27.240 - Yeah. 00:32:28.260 - And that is the like, we'll just figure it out. 00:32:31.380 Like we don't even want you to try 00:32:34.380 to establish any type of ontology or framework 00:32:38.780 or categorization or like any of the precursor steps 00:32:43.780 to perception where like we're sourcing and acquiring 00:32:48.820 and extracting signals from data, 00:32:52.180 just stop trying. 00:32:53.020 - Yeah. 00:32:54.020 - And so there are precursors to that, obviously, 00:32:57.940 like we have to be able to ingest and interpret data, 00:33:01.100 like humans do, because we've spent the last 40 years 00:33:06.100 trying to curate data for computers. 00:33:08.900 That's why we have JSON and XML and SQL databases 00:33:12.100 and everything else. 00:33:13.360 So really with what language models help us with 00:33:17.540 is the ability to comprehend data 00:33:20.180 that wasn't curated for computers. 00:33:23.860 That's really the big advance. 00:33:26.260 That does not make it AGI. 00:33:27.820 That doesn't mean it understands reasoning. 00:33:30.380 But it kind of like closed that gap for us 00:33:32.700 that now we can use all of this data 00:33:35.540 that was built for humans since time and memorial 00:33:39.180 without kind of like humans rebuilding it all 00:33:42.140 for the computer. 00:33:43.500 - Yeah, we talk about that a lot of like the value 00:33:46.340 of all this is getting the data, 00:33:48.580 making it more accessible. 00:33:49.980 - Right, so we have all this really, really good data, 00:33:52.460 but to your point, you asked someone to go organize 00:33:54.660 and catalog it so you can access it and-- 00:33:57.180 - We'll come back in 40 years. 00:33:58.340 - Yeah, more in a minute. 00:33:59.500 - In a minute, the way through, 00:34:00.340 they're gonna change the cataloging strategy 00:34:01.940 and then have to go do it, you know, 00:34:03.100 and it just, it never gets done. 00:34:04.740 And it's not sustainable to keep, up keep either. 00:34:08.860 That's why most companies will adopt. 00:34:10.540 We see it all the time, like, 00:34:11.780 they're gonna go get some big CRM or some other software, 00:34:15.980 and they're gonna say, this tool is great. 00:34:18.020 And then six months later, no one's using it. 00:34:19.900 Why? Why is it so uneasy? 00:34:21.020 - Yeah, because they just underestimate 00:34:23.020 like the amount of human toil that goes into 00:34:25.500 making that tool useful. 00:34:27.140 - Yes, and it's not gonna happen. 00:34:28.660 - So like, we're talking about like a sales force, right? 00:34:30.740 So sales force rolls out, they call it Einstein, I think. 00:34:34.060 So Einstein's are like AI chats, 00:34:36.020 so you guys are both laughing. 00:34:37.220 Are they a new client of yours that we shouldn't be? 00:34:39.580 - No, they're great. 00:34:40.500 - No. 00:34:41.340 - It's the sales force is awesome. 00:34:42.180 But yes, because it is such a powerful tool, 00:34:44.940 but it's only as powerful as people are able to leverage it. 00:34:47.980 - And then SAP, I saw a thing with this. 00:34:49.860 - But let's be clear, all these tools, 00:34:51.860 and this isn't to pick on Salesforce or SAP or anything else, 00:34:55.420 but these are systems of record. 00:34:57.740 These are not systems of execution. 00:35:00.740 What everybody wants is a system of execution. 00:35:03.020 - That's what I'm talking about. 00:35:04.380 - Right, so here's a fundamental question. 00:35:07.620 How many chatbots do you have to bolt onto a system 00:35:10.220 of record before it becomes a system of execution? 00:35:12.820 Like, let's put it in more realistic terms. 00:35:17.020 How many tires do you have to bolt on your car 00:35:18.940 before it comes to semi. 00:35:20.260 - Well, also-- - If you put 00:35:23.500 18 tires on your car, is it a semi? 00:35:25.820 (laughing) 00:35:27.940 Yeah, so like I don't know that there's a path 00:35:30.100 to getting from a system of record 00:35:32.700 to being a system of execution by just bolting chat 00:35:35.780 by talking about it. 00:35:37.100 Is the evolution then like we try that 00:35:39.900 and we're gonna realize that is not a path forward. 00:35:42.460 - We have a whole different generation of systems 00:35:44.660 that are coming out that are built to be systems 00:35:47.380 of the work execution. 00:35:49.140 And these are these like AI operating systems 00:35:53.180 for enterprises where they're pre-built to like, 00:35:56.500 do work, not record work. 00:35:59.100 - And that's something that is where like the layman, 00:36:04.100 when people think of AI and AG, 00:36:07.700 that's where people's brains really go. 00:36:10.340 It's like, run a diagnostic. 00:36:11.980 - Yeah. 00:36:12.820 - Run the diagnostic. 00:36:13.900 - Yeah. 00:36:14.740 - Return the report, right? 00:36:16.380 Give me this, and not give me this, do this. 00:36:19.980 - Yeah, for office. 00:36:21.740 - And actually, how about just do perform all that 00:36:24.060 without me asking, you just let me know. 00:36:26.700 - When the storm-- - When the storm 00:36:27.540 - And honestly comes up, you tell me. 00:36:29.140 - But the version of copilots or chatbots 00:36:31.820 we bolt on top of systems of record, 00:36:33.860 whether that's your one drive or your sales force, 00:36:36.860 it's, they're basically like a let me Google that for you. 00:36:40.140 - Yeah. 00:36:41.140 - They're not really, like they don't understand 00:36:43.380 how to perform work and they don't, 00:36:45.100 they're not objective oriented. 00:36:47.700 They have no, like, I hesitate to say they have no purpose 00:36:51.580 'cause in the casual vernacular, they do have a purpose. 00:36:54.500 - But that's the big difference between humans 00:36:56.820 and the LLM, right? 00:36:58.620 Is the human is asked to solve a problem. 00:37:00.660 The human goes and creates a plan to solve that problem, 00:37:03.060 uses their, not just the information that they know, 00:37:07.540 but their past memories of them solving 00:37:10.660 those types of problems in the past. 00:37:12.180 Like for example, I might call Brandon 00:37:14.420 And a certain thing happened where I might have said, 00:37:16.300 you know, last time I did that, 00:37:17.340 he didn't like that 'cause I didn't highlight the rules. 00:37:18.940 - You have context. 00:37:19.780 - But then I also figure out what tools I need to use 00:37:22.980 to solve that problem. 00:37:24.340 And I just don't, like that's, 00:37:25.900 I'm more of a systems integration. 00:37:27.500 Like how's all this gonna work together, right? 00:37:29.300 Like I might have a CRM, but like you just asked me 00:37:31.860 to go order flowers on the, 00:37:35.140 for the birthday of every single person that's a, 00:37:37.660 you know, blah, blah, you know what I mean? 00:37:39.180 - Salesforce doesn't happen. 00:37:40.660 So then that's, and I think that that's kind of the thing. 00:37:42.620 So like we got Einstein over here with our Salesforce data. 00:37:44.980 We got SAP over here with, you know, whatever they call theirs, 00:37:48.060 you know, I don't know. 00:37:49.220 But that, that, that they're all, 00:37:50.860 there's just huge blind spots. 00:37:52.740 - Yeah, they're an emotionally blind spot. 00:37:54.740 - But, I was right. 00:37:55.980 - But Sean also, they're not asking you to send flowers 00:37:58.900 on their birthday or whatever. 00:37:59.940 They're asking you to, to delight that customer 00:38:02.460 with a surprise on a special day to them, right? 00:38:04.500 And in order for them to, and maybe a level deeper 00:38:07.100 is in order for them to have some sort of loyalty 00:38:09.300 to your brain and come back and buy more, right? 00:38:11.540 But then we have to give those AI agents an objective. 00:38:15.040 And that objective is customer delight. 00:38:17.640 And then we have to give them a casual 00:38:19.840 or discreet deductive reasoning process that says, 00:38:23.300 this is how we delight people. 00:38:24.800 Like with this guy went to Notre Dame 00:38:26.320 and maybe he, you know, 00:38:28.460 and he looks like he spent some time in Colorado. 00:38:31.220 Maybe he'd want to go see the Colorado Notre Dame football game, 00:38:35.520 you know, in 2020, you know? 00:38:37.640 But here's the price of those tickets. 00:38:39.340 Yeah, right. 00:38:40.180 You know what I mean? 00:38:41.020 I don't even, maybe that's not even enough. 00:38:42.860 You know what I mean? 00:38:43.700 - But when we think of consciousness and awareness, 00:38:46.460 there's different layers of it. 00:38:48.940 And what you're talking about gets up in, 00:38:51.780 like what we would consider the directive layer, 00:38:54.500 of like where we have morals and values and priorities. 00:38:58.500 And the comprehension layer and the perception layers 00:39:02.620 are very low foundational layers of cognition. 00:39:07.060 And so like that's what we have to work on is 00:39:09.780 How do you take morals and values and memory 00:39:12.180 and inject them with comprehension 00:39:14.820 and then give them an objective to perform? 00:39:18.020 And that's really the situational awareness 00:39:20.680 that humans are able to come to 00:39:22.340 and then derive an action plan out of that 00:39:25.260 and then carry out those actions. 00:39:27.220 - The action plan and then carrying it out. 00:39:28.740 But I still think that maybe the easiest way 00:39:31.860 for that to get done is to create some sort of agent 00:39:34.900 that kind of lives and breathes with you personally 00:39:38.380 and understands you. 00:39:39.460 - Yeah, yeah. 00:39:40.300 - Yeah, Apple. 00:39:41.120 (laughing) 00:39:41.960 - It has memories about you. 00:39:43.120 - It has memories about you and nothing. 00:39:45.420 - It understands your moral system and your value system 00:39:49.160 because it's a personal experience. 00:39:51.960 - It is, yeah. 00:39:52.960 - And I don't know how that makes enterprises 00:39:54.520 a bajillion dollars, but you know, 00:39:56.560 it sure could be an easier way just 'cause it would flow 00:39:59.960 the way a life flows. 00:40:01.760 - Well, it makes them a bajillion dollars 00:40:03.920 because right now, whether you like it or not, 00:40:06.880 like we can go into human resources department 00:40:09.200 - I don't want to. 00:40:11.440 - People are expensive, man. 00:40:14.440 - Yeah, that's true. 00:40:15.280 - Right. 00:40:16.120 So one of the, so let's talk about, 00:40:20.760 if we have some people that are working 00:40:22.640 at large enterprises right now, 00:40:24.160 they're thinking about AI, 00:40:25.800 what should these companies, 00:40:27.280 what should these organizations be thinking about 00:40:29.680 in your humble opinion, 00:40:31.600 what should be the first things 00:40:32.800 that they should be trying to think through 00:40:34.760 to prepare for this, the uncoming slot, 00:40:38.120 on slot of AI. 00:40:40.120 Yeah, so obviously they need to be thinking about what is their organizational structure look like 00:40:47.040 in the future? 00:40:49.640 Like what does this do to teams and like leadership because you're gonna have a new 00:40:53.400 Like we don't even have to talk about like does this or does it not like decimate knowledge workers like move past that 00:41:00.200 But what does like the new manager look like once that happens when it's like I have to co-manage 00:41:07.280 some people and some agents. 00:41:10.360 - Right. 00:41:11.200 - Right? 00:41:12.040 Like what does that skill set look like 00:41:12.880 and how do you measure success and those kind of things? 00:41:15.360 - That's funny. 00:41:16.760 So like I don't think we've ever actually talked 00:41:18.840 about this before, but the managers who are used 00:41:22.440 to just managing humans are very well now going 00:41:25.480 to have to also be managing agents. 00:41:27.720 - Right. 00:41:28.560 - That is a whole new world. 00:41:31.080 - Sure is. 00:41:31.920 - That these people have never even considered. 00:41:33.600 - Yep. 00:41:34.440 - Well, because the agent, and I may be stretching this out, 00:41:37.080 But if we're talking about the agents are more successful 00:41:40.320 if they live and breathe as time goes on, 00:41:43.300 the agent kind of becomes its own entity. 00:41:46.680 - Yes. 00:41:48.000 Managing in the AI agent is more like managing a human 00:41:50.960 than it is managing a technology. 00:41:52.880 - Yeah. 00:41:53.720 - And then like when do you update its goals? 00:41:56.280 - Well, that's more really review. 00:41:59.200 - Well, but you're not that far off. 00:42:01.480 - Yeah, I mean seriously. 00:42:02.640 - Yeah. 00:42:03.480 - Yeah. 00:42:04.320 - Because you wouldn't want to do it every time 00:42:05.160 you're pissed off. 00:42:06.000 You know what I mean? 00:42:06.840 You'd want to do it. 00:42:07.660 You'd want to have a well reasoned, 00:42:10.880 powerful review. 00:42:11.720 You'd have to understand that Sean's a little upset right now. 00:42:14.320 You didn't mean you weren't mean to the ground. 00:42:16.800 And then do you get to the point where 00:42:20.040 that's where it gets a little scary, 00:42:21.760 where if agents really are successful 00:42:25.000 because they're being given like this runway to live on 00:42:29.720 because they're more valuable as they live on, 00:42:32.840 they wouldn't they start exhibiting self-preservation tactics? 00:42:37.840 - But potentially, but-- 00:42:42.600 - Which may or may not be bad. 00:42:44.800 - Yeah, I mean, that's not exactly how it worked, 00:42:48.320 but we could potentially, but then you've achieved 00:42:51.120 sentience at that point. 00:42:52.880 And we're talking-- 00:42:53.960 - I said I wanted to get here. 00:42:55.640 - But you can have AGI without sentience. 00:42:57.960 - Right, okay. 00:42:58.920 - Right, just because it can perform like a human, 00:43:01.480 - It doesn't mean it's self-aware. 00:43:03.240 - Right. - Yeah. 00:43:04.080 - And that's what, and just because it tells you 00:43:07.240 it's self-aware, doesn't mean that it is. 00:43:09.800 - Right, right, right. 00:43:10.720 It could be like, you know, I really wanna keep working here, 00:43:12.680 boss. 00:43:13.520 - I mean, you've met. 00:43:14.360 - There's still just a bunch of ones in the zeroes, dude. 00:43:15.200 - You've met people though, right? 00:43:16.680 How many people could you ask if they're self-aware? 00:43:18.840 And you'd be like, yeah, no, you're not. 00:43:20.480 (laughing) 00:43:21.880 - I mean, honestly, I interact with a lot of different types 00:43:25.840 of people in different roles. 00:43:27.000 And I will say that, I would say a good percentage 00:43:30.840 of people just seem to be just kind of a prompt. 00:43:34.380 And you can guarantee what they're gonna fucking say. 00:43:36.220 - Yeah, exactly. 00:43:37.060 Oh yeah, yeah. 00:43:37.980 Well, we joke about it. 00:43:38.820 We're like entire companies are a prompt for us. 00:43:41.740 We're like, that's one prompt. 00:43:43.180 That's one prompt. 00:43:44.020 - How many people do you know that guy just says 00:43:46.340 the same story over and over and over? 00:43:48.340 - It's like, dude, you've told me the story before. 00:43:51.500 Did you not bank that one? 00:43:52.660 And that you delivered that one, I mean? 00:43:54.300 - Yeah, fine, fine. 00:43:55.780 - Well, let's maybe let's go back 00:43:57.340 to some of those other things. 00:43:58.540 'Cause I think there's some more like present challenges, 00:44:01.420 like that's a little more forward looking. 00:44:04.420 First of all, is if you can't get your arms 00:44:05.900 around your data, don't try AI. 00:44:08.380 Like there's not, to me, there's not really any point 00:44:11.220 at that without integrating your organizational data. 00:44:15.700 It's just like a pretty cool version of auto-correct. 00:44:18.620 - Write me a poem. 00:44:19.700 - Yeah, write me a script or like you can do 00:44:22.300 what we would-- - Service level stuff. 00:44:23.780 - We would consider that task automation. 00:44:26.420 So if you look at like American Productivity and Quality Center, there's like tasks, activities, 00:44:32.360 processes, process groups. 00:44:34.520 And so what you're experiencing when you're like, you know, write me this code, that's 00:44:38.340 task automation. 00:44:39.340 Sure, right. 00:44:40.340 Well, you're not going to go any further than that. 00:44:42.780 You can really get to activity automation by integrating your organization's data to 00:44:48.280 it. 00:44:49.280 And that is not just assistive in nature where like maybe you're a little bit more productive, 00:44:55.460 that your employees can actually perform 00:44:57.220 at a higher level of expertise. 00:44:59.980 And so it extends their scope of practice. 00:45:03.660 And so we would consider that augmentative. 00:45:06.020 Like you're an augmented human at that point. 00:45:08.420 You can't just do more, 00:45:09.660 but you can do different and higher level things with that. 00:45:13.180 - And I see people who are already pretty expert level 00:45:16.980 in their field who use AI, 00:45:19.340 that it becomes an augmentation of that. 00:45:21.740 Even if they didn't integrate their, 00:45:23.100 I mean, they're integrating their knowledge 00:45:25.380 like what they know about it. 00:45:26.580 'Cause when they're writing their prompts or saying 00:45:28.380 what their starting point and their assumptions 00:45:30.540 and what the end result needs to be. 00:45:32.460 And they get fill in the blank sort of situation 00:45:34.380 with a more complete answer. 00:45:35.940 So you're saying like to do that at a scale, 00:45:38.380 company-wide scale, they have to have their data 00:45:41.180 and then at the activity level. 00:45:43.220 - Yep, because you can't expect everybody 00:45:45.580 and you probably shouldn't have everyone 00:45:47.180 writing their own prompts. 00:45:48.420 - Yeah. 00:45:49.420 - And so if you like in any given department 00:45:52.540 in an organization, you know, if there's 50 people 00:45:55.500 that do accounts receivable, do you really want 00:45:58.180 every one of those having 50 different versions 00:46:00.540 of accounts receivable prompts they're running? 00:46:03.140 - Well, yeah, you're right, it's not scalable either 00:46:05.180 to have 50 different people learn this new skill set 00:46:08.180 that you could literally just lift one layer up 00:46:10.940 and then have an expert who does that, you know, 00:46:13.380 or at the software. 00:46:14.220 - Well, and then that, like, we talk about it a bit, 00:46:16.300 like we're all taking on this whole, like, new wave 00:46:19.940 of Tecta in prompt engineering. 00:46:23.200 Because if you think about the Tecta of, 00:46:25.580 we installed servers and we haven't refreshed our hardware, 00:46:28.160 that cycle's like five to seven years or something. 00:46:31.480 The cycle of Tecta and depreciation in our prompt 00:46:34.880 is like a couple of months. 00:46:36.600 - Yeah, I think that's why people get so fatigued 00:46:38.860 and give up on it so quickly because it's like, 00:46:41.000 well shit, that's no longer useful. 00:46:42.900 Or that doesn't work as well as it used to work 00:46:44.880 four or five months ago. 00:46:45.720 - Right, because like a new model, 00:46:46.600 or they deprecated that model or the model drifted, 00:46:50.680 or there's a new set of rules the prompt needs to follow 00:46:53.440 and I don't have time to rewrite it. 00:46:55.840 So to me, that's like a body of knowledge thing 00:46:59.480 in symbolic reasoning. 00:47:01.400 And I think in a lot of organizations, 00:47:03.280 that's gonna be a service that they consume. 00:47:05.720 Because when you're just writing casual prompts, 00:47:09.640 it's fine to do it yourself. 00:47:11.640 But really, if you're writing finance or legal 00:47:14.480 or healthcare prompts, they should be written by an expert. 00:47:19.480 They should be validated by domain experts. 00:47:23.840 And then really they should kind of be locked 00:47:25.960 into that version so they can create repeatable results. 00:47:28.760 - Yeah, the locking into the version thing 00:47:31.840 is almost like a, it would be a piece of the audit 00:47:36.680 that they would need to go through every single time. 00:47:38.280 - Yeah, but saying one prompt version control system. 00:47:41.280 (laughing) 00:47:42.520 - Well, if I, so I come from an IT audit background, 00:47:46.280 I would be adding that to this. 00:47:48.040 - Like we do, we're just adding version control 00:47:50.760 to our whole cognitive studio, so you can be like, 00:47:53.360 it gets its own goo-ed that you can call via an API. 00:47:56.440 - Yeah, you're gonna need all that stuff. 00:47:58.600 - Did you, are we good? 00:47:59.440 - Because it has to be verifiable. 00:48:00.960 Like if I don't know what version of this thing went through, 00:48:05.240 right, it's, I can't place reliance on it. 00:48:08.520 - Nope, are we gonna blow right past this though, 00:48:10.800 as far as like from an educational standpoint. 00:48:12.900 'Cause I remember I had Mavis Beacon, 00:48:14.300 I learned how to type, right? 00:48:15.280 And that's a fundamental thing, 00:48:16.200 I still need to know how to type. 00:48:17.640 But like, are we gonna teach kids or add up to engineering? 00:48:20.400 - Yeah, something. 00:48:21.240 - It's just gonna be too-- 00:48:22.320 - It'll be too, it'll be ubiquitous. 00:48:23.880 - Well, but, but, so, and we've kind of talked about 00:48:27.000 this kind of idea of a cognitive, you know, 00:48:30.240 a cognitive team that is literally the people 00:48:33.480 who are gonna ultimately be the ones who are writing 00:48:36.040 the prompts, who are really trying to understand 00:48:38.240 the problem statement, trying to understand 00:48:40.040 - Yeah, we actually get it. 00:48:41.420 - Yeah, but do you think that those types of groups 00:48:44.580 start forming within these organizations? 00:48:47.180 - In the organization? 00:48:48.580 - Or is it more of an outside? 00:48:49.740 - I'm still like maybe. 00:48:51.540 - Right. 00:48:52.620 - Because typically when we put AI groups together, 00:48:54.620 they're like super concerned with like the nuts 00:48:57.060 and bolts and plumbing. 00:48:58.660 They're not that concerned with like the business application 00:49:01.500 and the reasoning processes behind it. 00:49:03.620 - They're still trying to figure out, 00:49:04.900 can we get our data? 00:49:05.740 - Yeah, that's more of like a psychology discipline 00:49:08.060 and it is a technology discipline. 00:49:10.780 Because it requires us, 00:49:13.020 so we'll sit down with groups of people 00:49:14.580 and we'll be like, okay, you perform a task. 00:49:17.620 Now, what we need to do is harvest that reasoning 00:49:20.980 or how that's performed 00:49:22.860 and create that neurosymbology, right? 00:49:25.420 Of we're gonna have an AI do it the same way you do it. 00:49:29.540 So that's a practice called metacognition, right? 00:49:31.860 Like we think about thinking, 00:49:33.500 and it's one of the hardest things for people to do. 00:49:36.860 And so in order to be a really good prompt engineer, 00:49:39.580 you have to think about how you think about performing a task. 00:49:44.220 And that's really difficult for people to do. 00:49:47.580 And it's a fundamental, it's a precursor 00:49:50.340 to good prompt engineer. 00:49:52.100 And so there's so many like base level skills 00:49:54.620 you would have to teach people that I don't see that thing 00:49:56.780 on practice. 00:49:57.620 - It's why most people I think give up on it too, 00:49:59.020 because it's so hard to sit. 00:50:00.620 And it took me a long time when I was started prompting 00:50:02.980 of like, you know, sitting in front of an empty screen 00:50:05.980 and being like, "Okay, how am I gonna ask this? 00:50:08.140 "What am I trying to get at here?" 00:50:10.860 And then you just, it's like learning how to Google 00:50:12.860 really well, but like another couple of layers up 00:50:15.060 where you're like, 00:50:15.900 - But harder. 00:50:16.740 - Yeah, yeah, yeah, yeah, 'cause like Google is more effective 00:50:18.580 when you know what the problem is already 00:50:20.060 and you can ask very good, you know, curated, detailed 00:50:23.060 questions. 00:50:23.900 Same thing, but like, way worse, or way more detailed, so. 00:50:26.900 - Yeah, and we see that in a lot of different ways 00:50:29.260 'cause there's even like that concept of attention 00:50:31.060 management. 00:50:32.260 And so what people wanna do is be like, 00:50:33.820 "Hey, I have 10,000 HR employee survey responses." 00:50:38.100 So what I should do is just boom them into a model 00:50:41.540 and be like, "Is it good?" 00:50:42.820 - Yeah. (laughs) 00:50:44.180 - Yeah, but like that's how this works 00:50:45.620 and they're like, "Why doesn't it know?" 00:50:47.140 - Yeah. 00:50:47.980 - Well really if you think about it, 00:50:49.220 like if I ask you that, like, 00:50:52.820 can you pay attention to all the survey results at one time? 00:50:55.580 You would be like, "No, of course not. 00:50:56.580 I can read one result at a time 00:50:58.860 and I can analyze its sentiment one at a time." 00:51:02.540 And then I would accrue all of that sentiment analysis 00:51:04.980 and I would post process that. 00:51:06.980 - Well, that's, isn't that where you would coach them? 00:51:09.300 You need a layer in between the raw data and the, yep. 00:51:14.300 - I did that with something in the NBA data. 00:51:16.100 - In the model, I mean. 00:51:17.020 - Remember we went through this exercise 00:51:18.340 where we had a bunch of NBA data and I was like, 00:51:20.060 all right, I got a bunch of data here. 00:51:21.340 I want to figure out who the best team is, 00:51:23.260 but I don't know how to define that. 00:51:25.020 Help me define that first. 00:51:26.740 Here's the data set. 00:51:27.780 And they were like, okay, you could do this, this, and this. 00:51:30.220 How does that sound? 00:51:31.060 And I made some adjustments. 00:51:32.020 And then I gave it the days that, 00:51:34.260 and it came back with a pretty decent result. 00:51:36.500 But if I just dumped that Excel file in there, 00:51:38.340 and I was like, who's the best? 00:51:39.740 It might have figured it out, but like-- 00:51:41.500 - But you may not even fit at all in the context, 00:51:43.820 when it may have truncated half of your data. 00:51:45.900 - Right, no. 00:51:46.740 - And you don't even know. 00:51:47.820 - Yeah. 00:51:48.660 - That's part of the misuse, I think, 00:51:50.740 at the enterprise level of these LMs. 00:51:52.540 - Oh yeah. 00:51:53.380 - I've walked into, well, I didn't walk into. 00:51:55.900 We had to help a company get prepared for AI, 00:52:01.060 and the board was like very, very concerned about AI. 00:52:04.860 We need to do something about AI. 00:52:05.980 First they were very bullish, let's do stuff. 00:52:07.500 And then it eventually turned to security. 00:52:10.500 We're gonna get hacked, you know? 00:52:11.820 'Cause they were on the board of a company 00:52:13.780 that had been hacked. 00:52:14.820 And what's just happening a lot, 00:52:16.020 had nothing to do with AI. 00:52:17.300 But like the first thing that they did 00:52:18.660 in this board meeting is they started googling themselves. 00:52:21.540 That was what they did. 00:52:24.420 - But like nobody had a bunch of boards 00:52:26.220 and we work with a bunch of them. 00:52:27.460 And it's just like they have a frickin' Ouija board. 00:52:29.740 They're just like, it said a, 00:52:32.300 we should totally do it, you guys. 00:52:34.900 - But like, you know what I mean? 00:52:36.820 Like there's some context that's involved in all this. 00:52:38.780 And I think it's like, what are you trying to accomplish? 00:52:42.180 And I think that really for, from what you're saying 00:52:45.340 is you're categorizing these different capabilities, 00:52:47.980 you know, whether it's reasoning or comprehension. 00:52:51.020 Like understand the tooling of what you're doing 00:52:53.060 'cause AI is not, AI is a very broad term. 00:52:55.540 - So broad. 00:52:57.260 and AGI even-- 00:52:59.420 - Even right, yeah. 00:53:00.740 - I mean, that's a broad, yeah, I mean, 00:53:02.260 I don't even know how to-- 00:53:03.100 - But that's where we just, 00:53:03.980 we don't think in most organizations, 00:53:05.860 like they need a consumable. 00:53:07.500 And they need to be able to validate the authenticity 00:53:12.540 and the accuracy of that consumable, but like beyond that, 00:53:16.380 I don't know that people like build this for themselves. 00:53:20.020 And so we have to have like the, 00:53:21.340 here's the nutrition label, right? 00:53:23.740 And that's what we work on, like you know what model was used, 00:53:25.820 what data was used. 00:53:27.020 But like trying to do it yourself, 00:53:29.620 and we were talking about this a little bit earlier, 00:53:32.820 like we work with huge companies. 00:53:34.820 And like two years ago, everyone was like, 00:53:36.780 "This is amazing." 00:53:38.140 And then they were like, 00:53:38.980 "Oh, no, this is kind of hard." 00:53:40.140 Like, but we're gonna like build all this stuff ourselves. 00:53:42.900 And now they're like, we flipped our bike over 00:53:44.660 and knocked our two front teeth out. 00:53:46.560 But also we got everybody like lathered up 00:53:50.460 about how great it was gonna be. 00:53:52.660 So what do we do now? 00:53:54.660 Yeah, how do we deliver something? 00:53:56.340 And the answer is like we have to massly deploy consumable, 00:53:59.860 like objective oriented artificial intelligence. 00:54:03.900 But that's like click a button, attach a file, 00:54:06.340 click a button, run a search, click a button and do nothing. 00:54:09.900 And it comes back to you with the right outputs. 00:54:13.340 And that's a harder nut to crack 00:54:15.100 than people want up to like 100%. 00:54:17.300 - Yeah, I'm still fine with my just casual problem 00:54:20.940 solving chat, GBT, dog, my day. 00:54:23.420 But I've unfortunately learned that that is just the absolute basic layer. 00:54:28.820 Yeah, exactly. 00:54:29.340 Tipper the ice rag this whole thing. 00:54:30.460 Exactly. 00:54:31.100 Yeah, but the biggest organizations are having trouble getting to the next layer. 00:54:35.460 So you should feel good about that. 00:54:36.980 Yeah. 00:54:37.100 Well, no, no, I mean, they really are. 00:54:38.660 Right. 00:54:38.860 Like there's the arms race. 00:54:40.100 Like the ones that do. 00:54:41.500 Yeah. 00:54:41.940 Well, well, so the ones that to your point earlier, and we had this podcast, 00:54:45.580 go back to the Aubrey and on podcast from last year, but don't touch this stuff. 00:54:50.020 If you don't have your data. 00:54:51.020 Exactly. 00:54:52.020 There's no point. 00:54:52.660 - There's no point. 00:54:53.500 - That's step one. 00:54:54.340 - If you can't get to your data, 00:54:55.640 there's no reason to spend time. 00:54:56.980 - And if you're looking at, 00:54:58.220 I mean, if you're looking at companies that be successful, 00:55:00.620 just talk to your friends. 00:55:02.380 Like if they're in IT, which companies 00:55:04.860 we're creating massive amounts of tech debt 00:55:07.500 and constantly just paying money to keep the lights on, 00:55:09.860 and which ones we're advancing and layering up. 00:55:12.060 'Cause those are the companies that are gonna be able 00:55:15.500 to even touch going past securely prompting chat GPT. 00:55:21.020 - All right, so Aaron, thank you. 00:55:22.860 If people wanna learn more about you and your business, 00:55:26.140 how do they do that? 00:55:27.020 - Well, they can certainly learn about our business. 00:55:28.700 I don't know about me. 00:55:30.020 (laughing) 00:55:31.980 You know, I'm gonna live in a cabin 00:55:33.260 and write a manifesto at some point. 00:55:35.300 (laughing) 00:55:36.140 - I'd love to read it. 00:55:37.700 - Yeah, I definitely go to just promptprivacy.com 00:55:40.420 and check it out. 00:55:41.420 - Awesome. 00:55:42.260 And it was designed by somebody that you know. 00:55:44.980 - Try not to knock anything over. 00:55:45.820 You say that. 00:55:46.660 - I'm trying not to break it. 00:55:47.500 Anyway, so this wraps up our, I think, 00:55:48.940 six episode of season two of the Big Cheese podcast. 00:55:51.660 So everybody, thank you very much. 00:55:53.260 Make sure to subscribe and like 00:55:54.780 and we will see you next weekend, week, week. 00:55:57.860 We'll see you next week. 00:55:58.700 - Yeah.