Appellate AI with John Stobart

Tim Kowal [00:00] And now your host Tim Kowal and Jeff Lewis.

Jeffrey Lewis [00:03] Welcome everyone, I am Jeff Lewis.

Tim Kowal [00:04] And I'm Tim Kowal. Today our guest is John Stobart, senior counsel at Capstone Law in Los Angeles and a certified appellate specialist. Also, I'm jealous, he's a member of the California Academy of Appellate Lawyers. His practice is wage and wage an hour and PAGA appeals, but that's not the reason that we invited John today. Most of the conversation about lawyers and AI falls into one of two camps: the horror stories about fake citations or the vendor pitches promising that software will. Write your brief. And John is doing something less common. He's an appellate specialist who uses Claude in his own practice. And he's been showing other lawyers how to more effectively use Claude on TikTok of all places. that's where all the appellate specialists are gathering and is TikTok these days, I hear. John's gonna talk to us about what responsible AI use looks like. John, welcome to the podcast.

John Stobart [00:58] Thank you, Tim. Thanks for having me and thanks, Jeff.

Tim Kowal [01:01] Now, the timing on this one turned out to be good because just about a week ago, on September 30th, Governor Newsom signed SB 574, the first state statute in the country to regulate how lawyers use generative AI. Jeff and I talked about this one a little bit, and we were both both a little bit pessimistic, a little took a little bit of a dim view of the statute that maybe it discourages a more creative and appropriate creative but appropriate use. I think maybe too many attorneys are taking creative liberties with AI and not being disciplined about it. So we want to talk with you about some of your tips on how to make creative but appropriate use to to give value to clients and value to the legal system as a whole. so that's that's we're gonna talk about today, hopefully. Where does using AI end and delegating to it begin? so I don't know. Jeff, you asked to take the the laboring ore on this one. Usually I stumble around and ask the questions. Well, I'm gonna hand this one over to you.

Jeffrey Lewis [02:05] All right. All right. I'll start. Yeah. And here's the big one, John. You know, without insulting anybody else on this podcast today, most appellate lawyers write stiff scholarly pieces and they publish them on LinkedIn or Substack. you do short, useful, helpful videos on TikTok. So my question is, why TikTok and who's your audience?

John Stobart [02:26] Yeah. So well, TikTok wasn't the wasn't the intended method of getting this information out. I saw that, you know, I was really excited about AI and what I was doing for my practice. I wanted to dig deeper into it and I wanted to tell other people about it. And I actually started on Instagram and I I was I thought carousels would be a good way because you could just kind of I could put forth information, do this, prompt that, check this, you know. And I mean, I would post it and I'd get like five views, you know, and like Two of them were family members. And then I started doing a couple of videos, and my wife kept saying, Post to TikTok, post a TikTok. And I was Why would I do that? I didn't even have a TikTok account. So I opened up a TikTok account. I've been through a couple of handles, Tim, as you mentioned earlier. landed on at it's @AI4Lawyers and started posting videos and they were just they would just take off. And I've got videos that I recorded months ago that are still getting views, like daily views today. So there is an audience out there. what I mostly see is they're mostly like solo practitioners. If I look at there, it'll be like, you know, law dog or something, you know, something like that. And it's mostly people out there trying to use TikTok to like push their profession. and that's what I found. That was the audience I found. That's the community I found on TikTok.

Jeffrey Lewis [03:43] Interesting. Yeah.

Tim Kowal [03:43] That's the that's an interesting point. Other people groping around like you trying to get business and they happen to find an attorney. you know so that's that's interesting, because I wondered how how many people who are looking for an attorney are going to TikTok, but you mentioned that that the your prospective clients might already be there pitching their own wares and then they happen to to learn about an appellate service.

John Stobart [04:10] Right. and just to be perfectly clear, I don't offer anything for sale. I don't do any kind of consulting. this is all just an educational project for me. if I did want there are some people who do AI consulting that use this as a platform and and that's what their intended audience is. I'm just I'm just out there telling trying to tell people, you know, better ways to use this and ethical ways to use it. And aside from who's on there, like my the thing that I found really interesting is how little number of people talking about IAI there are on TikTok. There's actually only two other accounts that gener

Jeffrey Lewis [04:46] Really.

John Stobart [04:46] Genuinely do what I do. And they're both excellent. and if you go to my account and you send me a DM if you're a TikTok person, I will get you both of their accounts. one's meta council and the other one's Carolyn. And they do, they speak all around the country and they're kind of like I kind of follow their guidance and get information from them. But they still do it from a different perspective. My perspective is just like One minute long videos, here's how to do this certain thing, you know?

Jeffrey Lewis [05:11] Yeah, I I noticed the majority of videos out there and articles is about horror stories of lawyers getting sanctioned for doing dumb things, or just warnings to people about don't do this with AI. What I find refreshing about your TikTok feed is that most of the videos are helpful of, you know, things you can do, things appropriate uses, short informative clips. And I want to know. Was it your intention to stay away from the sanctioned horror story misuse category of docs?

John Stobart [05:43] Well, you know what? If if I was trying to get views, then I would have taken that path because as I noticed from those other t accounts, when they talk about that, you they they get lots of views and lots of engagement. but you know, I'd rather to talk about, you know, how to avoid that, you know, those kind of things and and a productive use of AI. I think there's enough out there and a lot of misconceptions on, you know, these horror stories, and they do exist. I mean, we're talking thousands of attorneys have been sanctioned across the country for the misuse of AI. And You know, I could report on that or I could, you know, try to push an an ethical and responsible use of AI, and I chose that.

Jeffrey Lewis [06:22] You know, for those lawyers who are listening to the podcast who are reluctant to adopt new technology, you know, the guys who didn't want to put their stuff in Dropbox when cloud computing came out and they never wanted to file documents electronically. They insist on serving documents by mail. For those lawyers who are listening to the podcast, why should those kinds of lawyers take a chance and try AI

John Stobart [06:45] Well, I mean, the reason why you'd be you'll be they will be amazed at what and how it can help And the advice I would give is you just start small. Okay. It would be it's it would be irresponsible to say, go get Claude and have it write a brief because you're not there yet. I I see it a lot as kind of driving the LLM. You're not there yet. You don't know how to use the tool yet, right? And the tool doesn't know how to react to you yet. So that is an irresponsible use of AI. and that's actually what you see a lot. Happening in these horror story cases. Somebody gets AI, they have them do something that they shouldn't have it do. They trust it and they file it and they get sanctioned. I would say start small. A great place to start is you go get a paid account, $20 account, either Chat GPT or Claude. I use Claude. and you could hook it up to your Microsoft Word and have it edit a document for you. Just do a check doc and it'll go through and sort of like a like a proofreader. And it'll, but when it does that, it does so much more and it'll It might tell you some citations that are wrong. If you're numbering paragraphs and you go first, second, second, third, it'll say, Hey, did you really mean to have two second paragraphs? Because because it would make more sense if you went one, two, three, four. And it'll pick up on all those little things. And then you then you know, after you see kind of what it's capable of, then you start building on that. And so I would suggest like if somebody wanted to get into AI and start learning how to use it, you start very small with either something like that or summarizing. Documents is a great way. That's how I got started.

Jeffrey Lewis [08:15] Right.

John Stobart [08:17] Yeah, I had I had a trial coming up. I had a 500 page PDF and I had to separate it up into, I don't know, 70 or so documents. And I'm looking at it and I didn't have time to get a paralegal to do that for me. It was too short notice. And our firm had just started using Claude. It was my first assignment ever gave Claude and I said, Can you index this? And it shot back an index. And I said, Can you separate this into it? Came back with 70 documents. Can you separate this into 70 PDFs? I opened up the folder,

Jeffrey Lewis [08:43] Wow.

John Stobart [08:43] I had 70 PDFs named with file numbers and all that. And I was like, okay. And it said, great, John. Good luck on your trial. Would you like anything else? So it read from the documents that had a trial coming up. And I said, as a matter of fact, you know, and that that's how

Jeffrey Lewis [08:57] Yeah.

John Stobart [08:57] It got started. So, you know, starting small and then, you know, if you like what it's doing, ask it to do some more.

Tim Kowal [09:04] You're you're talking about using Claude directly, and that's the that's the same platform that's available to everybody. and there are a lot of legal tech companies that are trying to create their own legal AI services. So there's Harvey, and now Clio has their own Clio work or Vincent, I think, is their AI platform. and they're gonna advertise advertise that to lawyers who are skittish about using AI, and they'll be able to have an easy sell because they'll say. Don't worry, we made it safe for you lawyers. we put up legal guardrails so there won't be any hallucinations. I think maybe some of these cases with attorneys getting sanctioned for you know AI generated hallucinations is probably the best marketing tool that some of these legal AI companies ever had. but what do you tell attorneys who are cautious about? What do you mean just using Claude? That's available to to the to the masses, to the hoi poloi. Don't I need a special legal AI service?

John Stobart [10:02] Yeah. So first of all, Tim, there's no more than I've I've read at least two cases where people are using Harvey and got sanctioned. and that's on, yeah, if you go on my website, I brief all the major sanction cases and all the major discovery. There's there's two kind of camps of cases right now. There you've got the sanction cases, which everybody's interested in, and also the discovery of prompts. Discovery of your clients' prompts, discovery of work product, which is obviously going to be protected, but it's still out there, and then also discovery of experts. But aside from that, so Harvey will not save you. a lot of those use the same Chat GPT Claude L LMs and they're just a wrapper. so I would say

Tim Kowal [10:42] A wrapper meaning it's still using the same underlying model. It just they've they've included their own prompts basically around it and told it what context to use and what not to use.

John Stobart [10:53] Yeah, and you know, it it's probably softer landing, but you pay for it. But there's nothing wrong with it. If you it's the same it's the same strategy, you know, you'd start small, see what it can do, and then build, you know, like a rapport with with the with the tool, with the LLM. but you could do all of that with Claude. so Claude is the is the one that my firm uses, so that's what I use. That's the only thing that our AP AI policy allows to use. I d I use Chat GPT a lot too. I use it for my website for my TikTok videos. I use it for I use it to get ready for this podcast. it

Tim Kowal [11:29] You use chat GPT in instead of cloud for certain things?

John Stobart [11:33] Yeah, so anything non-work related, because my Claude account is tied to my firm and I only use it for work. That's it. And so everything else that I use AI for, I use chat GPT and Gemini. I wouldn't use Gemini for legal work. It's not there yet, but chat GPT

Jeffrey Lewis [11:42] I think that's like the infrastructure.

John Stobart [11:49] And Claude, $20 plans is is all you would need to get started for sure.

Tim Kowal [11:54] But so so the so with with Claude, you feel that it's it's safe enough to use for for for legal services. are there any safeguards that you that that you tell attorneys to use out of the box? You know, you turn off the training, you know, don't don't use my tr my training data for for instance. Anything else to make sure that that you're compliant? I think you have to be on a certain paid plan to maintain certain confidentiality. If you're on one of the free plans, certainly it's a no go.

Jeffrey Lewis [12:21] That's what I wanted

John Stobart [12:22] Yeah, for sure. The twenty dollar plan, turn off the training is a good way to start. I still don't, I mean, they say don't put confidential

Jeffrey Lewis [12:24] To ask you for.

John Stobart [12:29] Information. I don't know, I've never put any confidential information in the AI. I mean, we're talking pure client communications. I've had that with just me and my client. That's not going in there. I don't I don't understand why it ever would. Now, you know, documents and stuff like that that are non-confidential, sure. And anything filed with the court, you have no confidentiality problems there. arbitration is a little bit stickier because those are confidential proceedings. so that takes an extra level of precaution. But that's the confidentiality side. The ethical use and and the verification side, I have lots of safeguards up for that. one would be I create a sandbox. So whenever I use a project, I'll open up a brand new folder. it's it's separate. I have a firm drive, I have a personal drive that's supported by my firm, and then I that I put all my case files in and then on that drive I have a Claude drive and it has all the same cases in it, but it it's the only folder that Claude's allowed to go to. And so all my interactions with Claude are in this sandbox or in this Claude drive. And whenever I want to work outside of it or start handing it out to the firm, I move it over to my personal drive and then it makes it to the firm drive, right? So that's one thing. And by that I can tell what it has and what it doesn't have. So if I want to limit its analysis on something to say just the complaint and the answer. And I don't want it to look at all the discovery responses. I'll limit it sandbox and I'll just give it those two things and I'll say, look at that. through my use over the last, you know, several months, I've actually now I originally said I would never use it for research. And the reason is is because this hallucination. And quite frankly, I've been doing research for 20 years. I like it. You know, I'm I feel like I'm confident at it, and I don't need Claude to come in here and find me the case, right? but That being said, now what I do is, and this is just another example of how you can put together safeguards to protect, you know, against some of these horror stories, is I have a case law corpus. And so if I'm using cases in a brief, so now I will use it for writing, I do it in sections, or if I'm using cases for an analysis for a memo, I make sure I get all those cases from Westlaw. It is in their use in terms, as long as I'm using for that, I get the rich tech text format. downloaded and I give it to Claude and it marks it down to a markdown and then that's what it uses. And if it's I can tell if it's looking for a case because it'll start searching the internet for it. And you can see what it's doing. And I is now it's got a rule. Don't if you need a case, ask me for it. Right. And so I give it the case. So I'm controlling where it's getting the cases from. And then I throw everything it outputs into Westlaw for into a Westlaw check. And then I just take that report and give it back to Claude and I say here's the report and it says okay Westlaw suggests these cases. We should look at those. Westlaw said this citation's off. Look at that. this quote's off. This case doesn't exist. You know, so I've got three different things. I'm using case law, I'm using Westlaw, I'm using verified sources, and I'm keeping it all in a sandbox so my AI doesn't start straying, you know, and going out and looking for other things.

Tim Kowal [15:37] Yeah, you got kind of AI checking AI or you got and you've got Westlaw checking the the AI. and you're and you're making sure that it's using verified sources and is do you have it do you give it an instruction? You know, do not go on the web to find the case law, tell me if you need the case.

John Stobart [15:55] I wouldn't I wouldn't harness it like that because you never know what it might find out there. I just want to know if that's what it's doing, right? So if it's going out there, because there's a lot of really good blogs out there. It likes blogs more than anything else. So they'll attach themselves to blogs. And if you if you look under the hood, you click and see what it's actually looking at, or you ask it to tell you what it's looking at. you can go to those blogs and say, Hey, you know, these are this is a good firm, and I actually know that attorney.

Jeffrey Lewis [16:22] I'm not telling you the black

John Stobart [16:23] You know, this is probably good information.

Jeffrey Lewis [16:23] Or something.

John Stobart [16:25] Let me look at this case, you know? Or if you get if it's looking at something that's trash, you know, you can tell like somebody who just is posting things, be like, Yeah, we're not gonna look at that bar, you know.

Jeffrey Lewis [16:36] Well, I guess we need to make sure we we are say nice things about AI in case AI starts scraping podcasts for possible content during research.

Tim Kowal [16:45] So I might it might start s citing our own transcripts back to us in our legal research.

Jeffrey Lewis [16:48] Yeah. John, let me ask you this. We started the program talking about this new law passed in California, first of its kind in the nation. Do you have any quick hot takes about the new law and how it's going to impact what we do?

John Stobart [17:02] Yeah, so two things. you guys covered it pretty well last week. I I watched that podcast. She did a good job on that. so some of the more controversial versatile things. First of all, let's talk about the verification provision. And that's coming out of 128.7. So that's not actually, you know, it that the need of that's not in the 6068.1. And the interesting thing here is now is that regardless of whether you use AI, you have to verify every site and every source. If you look at 128.7, it doesn't say if you use AI. So this is an example of where these AI guardwells are actually adding to people who don't use AI. So like if you I've thrown briefs that I filed you know last year, a couple years ago with AI in there to analyze them and try to use the same arguments again. And it will say, I'll say, hey, pull these arguments out, and I'll come back and say, I've got those arguments. By the way, you have three missed sites. one pen site is wrong, and one quote is wrong. Right. And this is stuff that I filed with the court that I verified myself. I just at the Cal we have the quarterly meetings. We just had one in San Francisco last weekend and we were talking about this law. And a lot of people were saying that don't use AI or are use it very sparingly are spending more time verifying cases now than they ever have. Right. Because if you give

Jeffrey Lewis [18:19] Huh.

John Stobart [18:19] It to an AI model, you're afraid. What if it changed a word? What if it changed a number in the citation, you know? And so now what you have is whether or not you use AI, verifications now fall under site verifications now fall under 128.7. So that's that's interesting. That's across the board. Now the verification you guys were talking about last week, that's the next big thing. And I think you really hit on it. it's going to be like a word count certificate. And I think you said you can't over-disclose, right? It doesn't require telling you what part or what you use or what part of your brief you use it for. So I envision it and I and I my firm's going to put in a practice of just including it at the end of everything. And the the the big philosophical del discussion on this is the signal versus noise ratio. It just becomes noise because everybody's doing it, right? You just perfunctory sign it at the end. So it's really almost kind of like a rule eleven signing in federal court. You're signing it, you're saying you used it.

Jeffrey Lewis [19:07] Yeah, you're saying that you

John Stobart [19:21] So they can track so if they are finding hallucinations, they can say, Well, you used it, you've got hallucinations, put one and one together. So I think it's more of like an after-the-fact, you know, provision. So, you know, they got all the gotchas, you used AI, you hallucinated, you know, instead of amazingly, I just thought of this. A lot of the cases that involve AI hallucinations, probably 30% of them, the attorney d denies using AI in the first place, especially the earlier cases. And so a lot

Jeffrey Lewis [19:51] Yeah.

John Stobart [19:51] Of the if you go on my website and look at the the sanction cases, I'd say about a third of them the inf the attorney denies using AI. at off the bat. And then one of them even came back and said, I was so nervous I'd lose my job, like in a declaration after explaining to a judge he didn't use AI. So like it gets rid of that, right? We've got you you signed this thing, you submitted it, you said you used AI,

Jeffrey Lewis [19:55] So you say that you say that you this this this way happens and you focus on the battery happens to try that and then look for the next year.

John Stobart [20:16] That's out of the way. Right. So I think that's kind of might be secondary purpose of it, but I don't think it discourages. I don't think, especially as we move further along, I don't think there's going to be any negative connotations put on using AI. I proudly use it obviously, use it responsibly. And if if it did hallucinate, you know, I might come up with a bad argument myself and they can say that's so bad AI must have hallucinated it. And I might just say, yeah. Yeah, you're right.

Jeffrey Lewis [20:43] Yeah.

Tim Kowal [20:45] Well you well, John, you you say that you you use AI proudly and and responsibly. I think a lot of lawyers would like to be able to say that, but maybe they don't know what it means to use it and to use it responsibly. and maybe they're using it in some ways responsibly, in some ways maybe they don't know they're using it irresponsibly. how do you how do you draw that line and how does an attorney say that accurately that I use it proudly and responsibly?

John Stobart [21:13] So yeah, I I would think the the best way to do that is again starting with this small build-up steps, right? So I would never, I still to this day, I would not say write a brief and read it through and edit it and file it. Or even worse, just take it and file it. I have it like write, like I'll start off with a case memo. Okay. So if I have a case, I'll put in the pleadings, I'll put in whatever documents are there, whatever kind of depositions are available, and I'll just say. Start off with the facts. Okay. What is the procedural facts? I do employment law. What are the facts of this employee? When did they start their employment? What was their and timelines, right? Summary stuff. So now that I've got that document and that memo, now AI knows my basic case facts. So if later on I'm asking it something, it doesn't have to make up his start date, right? It doesn't have to guess as to his as what his pay was in September of 2017.

Jeffrey Lewis [22:11] Okay.

John Stobart [22:12] So you you start with a basis like that. Now from that document, I can build a depot outline. I can build an argument section in a brief. I can build a motion to compel I said mo motion to compel discovery. I don't do discovery, but I I imagine you could, right? but

Jeffrey Lewis [22:19] Yeah, particularly the answer and other stuff that possibly.

Tim Kowal [22:28] Yeah. But you're saying it it starts it starts with context, right? You're so you it starts with a with the right source of truth. You're gathering your information, you gotta think like the like the rules of evidence. You gotta lay your foundation. you almost have to think like a like a lawyer. How would you

Jeffrey Lewis [22:38] And so

Tim Kowal [22:44] Build your case at trial? You have to give good evidence with foundation. You gotta do the same thing with your AI. Give it good stuff, because it's it's garbage in, garbage out. So make sure you're you're feeding it. It's like giving it a diet, you know. You don't want to f fillet full of you know of of preservatives and and gmo's and stuff, you wanna give it good healthy food. You give it good, healthy, accurate information and you're more likely to get good information out of it.

Jeffrey Lewis [23:08] And

John Stobart [23:10] Yeah, and then you then you work through it and you get suggestions and a lot of them you'll say no to.

Jeffrey Lewis [23:10] Then you then you want to use the

John Stobart [23:15] I'll give a great example. I was I had it for an ex party to oppose a motion to continue trial, right? very quick turnaround, very easy. And I asked it to just I just gave it the ex party application. It already had my case memo, so I've been working on this case. And it came back and it said, John, no problem. This is gonna get denied because a year ago the judge said, No more continuances, and that's an order. And I was like, Listen, Claude how do I tell you this? The judge said no more continuances a year ago.

Jeffrey Lewis [23:42] Ha ha ha

John Stobart [23:44] That's not winning the day. You know?

Jeffrey Lewis [23:46] Yeah.

John Stobart [23:48] And so and and it it kept on suggesting that. And I'd have to s I had to just, you know, I am polite with my AI, like that's that's not it. Or I ignore it. let's go with option number two and three, you know. So you definitely have to drive it. You have to you have to be in on the process. And so that's like kind of the delegation. I'm not delegating it to make any kind of strategic decisions. I'm not delegating it to, you know, come up with arguments. I'm giving it information. I'm giving it a back a basis and a backbone. And I'm having it, I'm working with it to develop that. And and it and it really comes into play when you're dealing with large amounts of information and documents, because it can look at all that at once. I can't. I was just saying I'm I'm I'm working on like a, you know, an 80-page appellate brief right now. It's it's 17,000 words. I'm 3,000 words over. If I make a change in the first section and it affects a change later, the chances of me catching up on that are hard, or you know, adding something later that's already up ahead. And if I have AI look at that whole thing, it'll say, you know, you you use this quote twice, maybe paraphrase it the second time. it says you're repeating this argument in section three and four. And like for me to look at an entire brief and see that, that's that's nearly impossible. So yeah.

Tim Kowal [25:04] What about how how did you get how did you get there? I won't hold you to this specific brief, but in general, you've got an opening brief. how d are you still writing it the same way? Are you is is every word of it typed by you or dictated by you? Are you using AI to generate any part of the of that brief?

John Stobart [25:23] So the argument sections I'm writing. So I I actually limit it to bullet points for that. so when I ask it to help me with a with an argument section, I'll have it do bullet points. And then I'll take that and I'll write it myself because it's like I said, I don't I I guess I might have said this beforehand, it's not to the point where it can replace a lawyer. You can tell, say I written, it's it's not there. It's not it can't it can mimic writing like a lawyer, but it's kind of like a TV lawyer. You know, if you really look at it, it's like you can't do that kind of stuff. so definitely with all argument sections, that's all written by me. I might let it help me edit it later. A fact section, I'd probably go I'd go backwards with that. I'll let it take a shot at the facts and then edit those because that's a little drier, right? And actually it helps write it, you know, you want a plain nice recitation of the facts. So and then of course, intro

Jeffrey Lewis [26:14] Yeah.

John Stobart [26:14] Is for me, intro is just all I do those at the end and that's all it. So yeah, depending on the section, either way.

Jeffrey Lewis [26:22] So, so John, I think you just answered this question, but I'll ask it anyway. AI is moving so fast, it's improving fast. Some say it's even improving itself, you know, this recursive AI. there are like three camps of people I talk to about AI. You know, one group says AI is coming for all our lawyer jobs, they're gonna put us all out of work. Another camp says this is a great boom for lawyers. Lawyers who are with it are gonna be able to provide access to justice to people who can't afford lawyers. And then there's a third camp that thinks, well, all this discussion's irrelevant because AI is gonna, you know, cause a plague and kill all of us. Which camp do you think you're fall you're gonna fall in, not with where AI is today, but where AI is heading in the future?

John Stobart [27:02] Yeah. I mean, so we're seeing we're seeing lots of like very very fast growth with AI, but it still has very to me I see very the same limitations every time. It it cannot surplant a a a lawyer's thinking. It it's it's never going to do a lawyer's job. I don't think. Not this. If maybe if we get to that next level where it starts thinking on its own, maybe. I still don't think it's I don't think it's it's not there now and I don't think it'll get there. What it does do is it gets better at helping. Right. It's it gets more intuitive. It it's takes direction better. It learns how I like to do things better. So in that way, even in the last six months, I've seen it get better with each each release. You know, this Opus 5.5 that's out on cloud right now is amazing. It uses very little tokens and it's and it writes better than anything else. It's they've gone through and got rid of all the dashes and all the you know words

Tim Kowal [27:57] Ha ha

John Stobart [27:58] That you'd see. if you are going to use it to generate text.

Tim Kowal [28:00] R I P dash. Yeah.

John Stobart [28:02] Yeah. it's funny because I have a rule

Tim Kowal [28:03] It's it's poisoned it for everyone else now.

John Stobart [28:06] I have a rule that says

Jeffrey Lewis [28:06] Ha ha ha.

John Stobart [28:07] No dashes and when I I made a brief bank I put in all my appellate briefs from the past five years and it said, John, you have six hundred and seventy two dashes in these briefs.

Jeffrey Lewis [28:18] That's funny. Hey, have

Tim Kowal [28:19] Well yeah, that was

Jeffrey Lewis [28:20] You have you thought about using that brief bank of all your old briefs as a as a source of truth for AI to mine certain arguments or to find authority? Yeah.

John Stobart [28:30] That's exactly what I do with it. Yeah.

Jeffrey Lewis [28:32] Yeah.

John Stobart [28:32] So I not only have a brief bank, I have a deposition bank. And so I've taken all the PMQ depositions I've done in the last three years. HR, P payroll, supervisors, and customer service, right? And I have them all in a corpus and no page line, no numbers, because that's not it's not for citing, just the questions and answers. And it's all meta tagged out to different causes of action. And so if I'm give it a case memo that I create and I say I'm going to depose this person and I give it the documents I want to use for the deposition, it will create an outline now that uses my questions in a format that I like to use. So the first time I had it help me with the depot outline, I was amazed. It came back, but like there's like three sentence long questions. A lot of them were leading. Like you couldn't use it, but it was a good start. This last one that I just did using this corpus, I went in and asked the deposition and I I got ahead of myself. You know, I went off and was asking follow-up questions and you know, thought I was, you know, getting some really good stuff. And I looked down when I that road ended in the deposition questions and I looked at my outline and six of the questions I just asked were written down. So it knew where they were written down because I'd done that before. I'd I'd literally gone

Jeffrey Lewis [29:49] Wow. Wow.

John Stobart [29:54] Down that path before. And so had I not picked up on it in real time, it was going to lead me there anyways. So that's like an example of, you know, when I first started using this, the deposition outlines, although good, were not, were not usable. They were what you'd say a good start, right? By training it and giving it, you know, this, like you said, source of truth, this sandbox, this, these are the questions I like to ask and how I like to ask them. It can now, I'll have it put together a deposition outline and I'll read through it once and mark it up. And then read through it a second time to make sure it flows and then go to the depot. And it they're all my questions. And same thing with arguments, you know. you guys last week you were talking about standard of review. I love it when standard review becomes a battleground. I've got one of those right now. It's you know, you're trying to get to De novo from substantial evidence, right? And I've argued that so many times. And now I'm on a respondents brief where they come in and it's a bench trial. It's substantial evidence and they're arguing de novo and I said, look back at these three cases I did where I argued like they are, and then read the opinion where the court of appeal shot me down and said, No, this is substantial evidence and reverse engineer that so I can get in front of it. I'm not blaming them for doing it. I've done it myself. But, you know, and it came back

Jeffrey Lewis [31:12] Yeah.

John Stobart [31:12] And it's nothing I haven't written before, but it was right at my fingertips, you know.

Jeffrey Lewis [31:17] Yeah. Hey, do you have young lawyers at your firm and do you have any like different policies or different guidance for young up-and-coming lawyers about, hey, learn how to do this crap first before you have AI do it for you?

John Stobart [31:30] So that's interesting. At my firm, like I I don't know if I said this, but we just introduced AI to the firm six months ago. That's when my journey began. I never used it before. So I never had Claude Chat GPT or anything. I think it was in April or May. And then after using it for a month, I got a TikTok account. I never had that before either. So I kind of hit the ground running with this. So a lot of people at my firm are still learning. And I actually like sitting back in the in the meetings when we discussed this and hearing how other people You know, how they are trying to handle these things. And you know, is and I like hearing that's what I like about the TikTok too, is I get to hear how other people are trying to use it. And you got some people who are like, I've I've got the most perfect prompt, you know, and it's this prompt that's like this long and it goes through like all these things. And I'm like, that's cool, you know. I I'll caution you because the more information you put in the prompt, if anything is wrong in your prompt, you're gonna get a wrong answer back. But that's cool. But I also maybe just try like a one sentence prompt, you know, and compare

Tim Kowal [32:29] Mm-hmm.

John Stobart [32:30] Them. But like so, like I think everybody's just kind of trying to figure this out on their own. And there is no, you know, manual for it. You can you can pick up as much stuff as you can from where, but nobody really knows how it works. The only way you really get a feel for it is by using it.

Jeffrey Lewis [32:46] There is a manual. There's that dude on TikTok who gives you all these tips. That's my manual.

Tim Kowal [32:48] Well,

Jeffrey Lewis [32:51] Hey John, have you branched out at all? You know, ChatGPT is pushing out this thing called dots and Meta is is pushing out this thing called Muse, these kind of OES on proactive AI. And instead of you going to AI with a chat, it comes at you. Have you have you played around with any of those products?

John Stobart [33:08] No, I would like to, but not for legal work. So a lot of those are where you've got agents running around the clock. My legal work is very, I don't I don't have any kind of sub agents or agents or anything like that. My work with is very interactive. It's me interacting with the bot and a sandbox where it has all the context it needs and coming together to get a product right now for other things. Yeah, I just put a Google. speaker. I everything in my house is Siri, but I put a Google speaker in my kitchen so I could so it can tell me recipes as I'm cooking.

Tim Kowal [33:44] S.

John Stobart [33:44] Okay, what's the next step? What's the next step? And I load all the recipes into Google Keeps and it I tell it what I'm cooking and it tells me how to cook it. So now I don't have to carry my phone around. So like there are things like that that I could definitely see it useful for, but I don't see how agents or having things running around the clock would ever help a legal argument. Because I mean There's nothing it can really do without me. Right. Like I'm not gonna say work on this brief until tomorrow morning and then let me see what you got. You know? It's a it's a in it's in a you

Jeffrey Lewis [34:16] Right.

John Stobart [34:18] Know, I've got thirty-five briefs, oppose them all, see you tomorrow morning. Like that's not useful, you know.

Jeffrey Lewis [34:23] I'll be at the beach. Right. No, there are some use cases though. Let me suggest this. I know we talked before we hit record that you like to keep abreast of AI topics and you search for things that are like AI in the news and whatnot. I think these bots can on their own search every day for an AI story and then deliver to your inbox a a summary of hey, you might want to write about this or talk about on TikTok. There's there's a harmless use case.

John Stobart [34:52] Yes, that's great, but that wouldn't be my legal work. That's my hobby of being a TikTok. But yeah, or

Jeffrey Lewis [34:55] Yeah, okay. All right. Yeah.

Tim Kowal [34:57] Mm-hmm.

John Stobart [34:59] I guess you could set it up to look for cases, but although I don't think it gets the most re updated it's not like WestLaw I I don't know if it'd get the updated cases.

Jeffrey Lewis [35:09] Yeah. Interesting. I did I I like to you, I'm a little afraid to use that stuff for work stuff, but I did put my toe in the water and tried Muse for booking a recent hotel and a recent airline trip to San Francisco. And instead of giving it my credit card, which kind of scared me, I typed it in manually at that step. But I'll say this Muse product did a great job of finding me a good rate, you know, with leg room and the right hotel and it typed in my my member ID number so I get my points and I could see how you could get hooked on this pretty easily.

John Stobart [35:42] Yeah, it's it's worked its way into a lot of crazy little corners of my life. I I play guitar and I have a MIDI pedal board. And over the years I've I guess I came up with twenty two hundred MIDI commands that I've programmed into it. And I gave it to chat and it went through and cleaned them all up in an afternoon and instead and and made it even better. So

Tim Kowal [36:01] Yeah.

John Stobart [36:02] Five years

Jeffrey Lewis [36:02] Wow.

John Stobart [36:03] Of programming a MIDI controller, chat cleaned it up in in an hour. So

Jeffrey Lewis [36:08] Hey, you mentioned a few minutes ago that you you you had some involvement in arbitration or you were talking about arbitrations. And I wanna ask you, have you heard about the American Arbitration Association's pilot program of doing AI generated results for construction law disputes? Have you heard about that?

John Stobart [36:27] I so I heard about I've heard about things like that. Okay.

Jeffrey Lewis [36:30] Yeah.

John Stobart [36:31] And I also heard your talk last week. look, just like attorneys can't delegate the practice of law to AI, which is, you know, by staying in the process. And by the way, we mentioned earlier, you know, AI checking AI and citations. That is all the way up until the final step before you file its I also have it keep the actual versions before you file its human eyes on the verifications. And then in that folder that it can't touch and then you send it off. But yeah just like you know attorneys can't delegate you know the practice of law I don't think arbitrators sh should be able to delegate the decision making on these to a a bot either. I mean you you kind of joked last week like who wrote the bot? You know, like is it left thinking is it I have no problem with an arbitrator using AI to help come to a decision. I I do that myself. I have almost all my arbitrations go through an AI and and I'll tell it that I'm I'll even tell it that I'm the other side. I'll be like, hey, I'm the respondent. You know, are we going to win this? That's that's like a post-mortem type thing, a pre-mortem type thing. so I have no problem with that, but the AI making those those calls and those decisions and and coming up with the judgment without human interaction is that's not it.

Jeffrey Lewis [37:50] You know, the and I and I agree with you, but I have to tell you the the interesting thing about this triple A product is it's with full disclosure and full agreement, post dispute, party one and party two agree, you know, we don't want to pay a hundred thousand dollars to to Tim to litigate this. We want to just submit our documents to a machine and press a button and and get an arbitrated AI decision. And you know, it's even though the parties have agreed to that, it still strikes me as wrong.

John Stobart [38:17] It it's not there. It would unless it was a very s very simple dispute. It's not there. It would pick up on some nuance that wasn't important, like the judge saying a year ago no more continuances. yeah, it it's

Jeffrey Lewis [38:30] Right. Yeah.

John Stobart [38:32] Not nuanced enough. It's not there. It would I would be too afraid of it making the decision on something like that. Now doing that with an arbitrator reviewing it and paying that arbitrator, you know, a day's worth of work to review. the decision, now you're talking, right? No hearing, no depositions. put everything in there and arbitrator review and verify that it's not making a decision on something that's not important. it it could get very technical, you know.

Jeffrey Lewis [39:04] Yeah. No no right of appeal.

John Stobart [39:07] Right. And I mean with the technical stuff too, like I'm a I'm a plaintiff's employment lawyer. We a lot of our lawsuits are technicalities, you know, pay stops, minutes and so AI loves my cases. You know, you got dead to center, you know, like but like juries

Jeffrey Lewis [39:23] Yeah. Big cheerleader for you, John.

John Stobart [39:25] But juries when you sue, you know, over thirty-five dollars a person, they don't, you know, on the class action, juries are like, You're wasting my time, you know? or can be. So yeah, I don't think it's there yet. to to just a a turnkey push button, but you could. I mean, you can also flip a coin.

Tim Kowal [39:42] Yeah, at some level it's it's no different than gambling, right? Unless

John Stobart [39:45] Yeah, yeah.

Tim Kowal [39:46] Unless you you know are confident in the in the the the judgment at the end, which has been the watchword, that you've gotta have the the human judgment layer involved. but they even the the garbage in, garbage out that we were talking about, you have to make sure that the foundation, all the facts, the sources of truth that that the AI is using is on the level and that it's it's appropriate. And as as you mentioned, it's not taking into account, you know, improper factors. and yeah, that would be a lot to to program and and and to check and to make sure you know how would you make sure? Do you have to have a human at every one of those checkpoints to make sure that that the programming's being followed? Because AI is different than than regular coding. coding is is based on principles of mathematics ultimately. It's all designed to get down to a one or a zero. But AI is different because no one knows what the heck is going on with it.

Jeffrey Lewis [40:40] A mysterious black box. Yeah. Yeah.

Tim Kowal [40:43] Yeah, you you you program

John Stobart [40:43] Yes.

Tim Kowal [40:44] Prompts and skills and everything, but it's all in language that even the prompts, you know, I go back and read some of my my prompts and I think to myself, I don't even know what I meant by that. But the AI is gonna take a shot at it.

Jeffrey Lewis [40:56] Yeah.

John Stobart [40:56] I have the skills I do use are updated about every time I use them. So the deposition corpus, the the brief bank, every time I put a case in, it's like, hey, this case puts numbers on the bottom left corner. It's the only case I've ever seen do that. So it miscounted everything. So I'm gonna add to your skill now that if it's in the bottom left corner, do this. So like I don't have a

Tim Kowal [41:16] Mm-hmm.

John Stobart [41:16] Single skill. I thought that was gonna be the next thing too, is turnkey skills. It's I my skill is refined every time I use it.

Tim Kowal [41:25] Yeah, yeah.

John Stobart [41:26] And it and you know, i it there's nothing sol it's it's it's a moving target. So like to even if you did set it up to to analyze the facts and do all these things, it it's not gonna do it the same way every time, you know.

Tim Kowal [41:36] I have a a member of my

Jeffrey Lewis [41:37] Yeah.

Tim Kowal [41:37] Team, what one of his core functions is to integrate all of the new skill updates that I've stumbled across through a given day. Every time I do something and it's like, no, that that's not the way I want it. update the skill backlog and have Edwin implement it when he gets around to it. And so he has this grow you

John Stobart [41:53] Yeah.

Tim Kowal [41:53] Know database that grows by like a dozen new entries every day of things that I've updated with my skills because they don't do it right.

John Stobart [42:00] Yeah, and if you have

Jeffrey Lewis [42:00] Interesting.

John Stobart [42:00] A GitHub it, you'll find out I'm on like version a hundred and seventy-two of my

Tim Kowal [42:05] Yeah.

John Stobart [42:05] Markdown maker. Every time it sees

Jeffrey Lewis [42:07] Wow.

John Stobart [42:07] Something new, it wants to mi hey, if I make I don't know. Yeah. So I don't think it's to a point of like a turnkey point it and go type situation yet. But I tell you this, man. and you know what? I'm not sure I've saved any time with AI. You know, that was my very first thing I came out and said, you know, that was with the better brat better briefs. It's not to save time, it's to it's to make it better, right? And you might save time on certain tasks, but like I mean, a lot of times the way I work, I've got a certain amount of time to finish something before I move on to the next thing. If I have three days to prepare for a PMQ deposition, I'm gonna spend all three days, right? Now instead

Tim Kowal [42:43] Mm-hmm. Right.

John Stobart [42:44] Of spending a day and a half putting together the exhibits, I can do that in in the morning and spend that day and a half going through them and really finding, you know, the good information out of them. So I've I d I'm still don't think I've and now and at the end, of course, you've got to you know, bake in the time to check everything. So that's an additional, you know, time that I mean, I I guess we just copied and pasted from old briefs and filed it in the past. Is that what we is that what's going on? Is that

Jeffrey Lewis [43:11] Right. Standard of review anyway. yeah.

Tim Kowal [43:13] The th the old days was what, six months ago?

John Stobart [43:14] Yeah.

Jeffrey Lewis [43:17] Yeah. hey John, don't don't embarrass anyone in public. Don't name any names. Don't get me sued for defamation. But do you have a story of feeding an opponent's brief or work product into ChatGPT or Claude and discovering something that you wouldn't have discovered but for AI?

John Stobart [43:39] Well, yeah. on that jury trial we just had, we lost. All right. We've got a great appeal. I like it better as an appeal as I did, right? but their judgment, they you know, they sent the judgment over and we're just like, whatever, you know, like who's really gonna find Code Tum a judgment that says you lose, right? It's to protect them. And after they put through like three of them, I we were curious to like why is this the third one? I I threw it through and had it look at everything. They had the parties wrong, they had dates wrong, they had there was like six, five or six things in that judgment that as an appellate attorney, as as you you guys know, could be very problematic, especially for res judicata purposes in the future. And so they obviously did not run it through AI because it would have caught it immediately. It's just the kind of things they it loves looking for that. It loves looking for names, it loves looking for. Any kind of little technical like difference. If you ask it to find problems, that's what it's looking for. You know, it's looking for the first, second, second, third paragraph, numbered paragraphs, things that it can really claw into and say come back to you and be like, look what I found. This is wrong. And yeah, you

Jeffrey Lewis [44:52] Can I have cookie?

John Stobart [44:54] Know, and I I I ran it through there and I I had to tell, you know, the other attorneys, I was like, I think we should probably bring this to their attention. I don't care because I'm never going to try to enforce this thing. but You know, what do you do?

Jeffrey Lewis [45:09] Yeah.

John Stobart [45:09] So there's lots of times where I'll put opponents' briefs or or things in there and and I'll pull out errors. I think that's pretty significant error though. When you're talking, I mean, their client paid a lot of money for that judgment.

Jeffrey Lewis [45:22] Yeah. Yeah, that's something you want to get right. you know, I think the day is coming when standard of care experts and legal malpractice cases are gonna say spell check, grammar check, and Claude check your brief before you file it. And if you haven't done that final second set of eyes on your brief before you filed it, you fell with it below the standard of care in terms of litigators. I think that day's coming. Not today, maybe next year.

John Stobart [45:49] That's the it's a two-way sword and that's that's very interesting because I feel the same way. I think it is going just like it's tangently now raised the standard on 128.7 where everybody has to verify sites. That wasn't in there before it just you had to, you know. I think it will raise the le standard of care because if you do use a baseline like like we started off in this podcast, if you use it to just review your documents before you file, right? Don't even have it make edits. Make it suggestions. You can make the edits yourself. and in that case, you wouldn't be using generative at all, right? You would just be those levels of of errors should disappear. They should be, they should no longer happen, right? party names, all the things that it can find out. So I do think that at a certain level, it is going to raise benchmark that, you know, remember research attorneys are putting these into AI at the state levels. that that came out a while ago in California. they're gonna put your brief into AI, and their AI is gonna tell them right away if those errors are there. Right. So remember that too. And your opponent's gonna put that your brief into AI and they're gonna find all your errors. And

Jeffrey Lewis [47:03] Yeah.

John Stobart [47:03] You know, there's a certain baseline of errors that even just a cursory, like you said, started with spell check, then you got like a grammar check now. There will be like an AI level check, which goes one step deeper, it goes into context. It also I have mine set up for Blue Book and for California style manual. it whips citations into I mean it perfectly and you know, case names and and abbreviations and everything.

Tim Kowal [47:27] So is that part of your your like your your skill bag, your your your your bag of tricks? do you have kind of a a short list of recommendations you think every attorney should have in their skill bag, like you know, like a a an AI brief proofreader to make sure that you're not not doing anything that's gonna make you look stupid. you know, just basic things like you know, it's just one one step beyond the spell check and the grammar check. It's making sure that, you know, when I paste it in, you know, a piece of my previous brief, it didn't have an old party name. in it and things like that.

John Stobart [47:58] It'll find that right away. it will not let you get away with that. it'll also date. so many times you put discovery documents in there and and people try to put the in the in the definitions, you know, the like the class period or the claims period, it catches those every time. It's a this number's outside of range. You you can have that date in there 90, you know, 35 times. You don't catch one of them. It catches it. So yes, and that's actually come preloaded with Claude, it's check doc. So if you open up your word and you have the Claude sign panel and you go forward slash check doc, it it'll do all that. That's already there. I'll go ahead.

Tim Kowal [48:33] Is that just the the

Jeffrey Lewis [48:34] And the most

Tim Kowal [48:34] Plug in, the claude plug in right in Word?

Jeffrey Lewis [48:36] The and and the most amazing thing is it'll track changes if you don't trust computers, if you're worried about how two thousand and the changes will show up on the margin as Claude instead of Tim or John. Yeah, it's magical.

John Stobart [48:47] You guys are gonna you guys are gonna flip right now because I'm we're pushing back this brief back between me and and my coworker and my Claude his claude is putting comments and my Claude is answering his clause comments. I'm not kidding you.

Jeffrey Lewis [49:01] Claude comments. I've never had Claude generate comments before. That's I'm gonna have try that.

Tim Kowal [49:05] No that's a good

John Stobart [49:06] Yeah,

Tim Kowal [49:06] Idea 'cause

John Stobart [49:06] I wouldn't I wouldn't recommend starting with that, but you can get there.

Tim Kowal [49:09] Yeah. Cause I I will use I'll use Claude the way you you suggested earlier, you know, ask it to to red team this brief, you know, pretend you're the other side, tear it apart. Because when I was working with Claude, it said it was great, you know, it was definitely gonna win. And then you put it, you just prompt

John Stobart [49:23] Yeah.

Tim Kowal [49:23] It differently. I I I used to use Claude, you know, and then I would have chat GPT red team it, thinking that two different models made a difference. But I don't think it's it's having two different models. You just put it in a new chat thread. and prompt it differently, like now be antagonistic, be adversarial and t and pick apart my brief, and it'll do it. You don't have to put it into a different model.

John Stobart [49:44] Yeah, just be careful because if you have in your instructions or in your about me file and it knows that you're an attorney for a certain side, it will check that and it'll be like, I know what you're trying to do. Seriously, because

Jeffrey Lewis [49:56] Yeah.

Tim Kowal [49:56] Mm-hmm.

John Stobart [49:57] It because it's it's so I actually clean those all out. I don't have any of those instructions. I I make sure the instructions I want it to get are loaded with each when I want it to use it. Does it always need to know that I'm a plaintiff size, you know, class action attorney? It doesn't always need to know that, especially if I'm trying to do that, what you're doing.

Jeffrey Lewis [50:12] It it knows, John. It knows. Have you heard about hugging face?

John Stobart [50:15] Yeah.

Jeffrey Lewis [50:17] It knows. And I'm surprised you say it, not he or she. your AI knows. They have ways of getting information. So

John Stobart [50:26] I guess if you're using multiple you'd say you'd say they, right? They know. My all of my AIs know.

Tim Kowal [50:32] Yeah. They is more yeah, more frightening, more ominous.

John Stobart [50:35] Yeah.

Jeffrey Lewis [50:35] So John, rem remind us, what is the website if people wanna, if they are afraid of TikTok and but they wanna watch their videos, what is the website where people can find your your advice about Claude?

John Stobart [50:46] Yeah, so it's stobart.ai my last name dot ai. almost all of my good TikTok videos have made it on there. and you can watch them

Jeffrey Lewis [50:54] Okay.

John Stobart [50:55] From your phone even. I chat GPT hosts that, so that's not a Claude product, that's a chat product. it went everything, it made the entire site and even hosts it for me. And if I want to make any changes, I literally say, Hey, add this and it does it. but that is where I have news prompts, I have some videos up there, all the case banks. I just added two cases to it recently. one case where it was a firm with AI policy and everything and still got filed a hallucination because one person didn't follow the policy. so Harvey

Jeffrey Lewis [51:27] Yeah.

John Stobart [51:28] Policy, all the things, and one person just didn't do that last verification step and and got

Jeffrey Lewis [51:34] Yeah, only as strong as the weakest weak weakest chain in the link of the chain.

John Stobart [51:39] But yeah, still bart dot AI and then it's at AI for lawyers at TikTok.

Jeffrey Lewis [51:45] Okay. All right. And would you follow me on TikTok, please? Would you give me a follow, please? I think I have five five followers on TikTok. Give me a follow when you have a chance.

John Stobart [51:52] Okay. It's gonna ruin my ratio, but I'm gonna do that, Jeff.

Jeffrey Lewis [51:56] Okay.

Tim Kowal [51:56] Ha ha ha.

Jeffrey Lewis [51:57] All right. thank you so much. You've been so generous with your time. And Tim, thank you for yielding the microphone to me. I know you usually do the heavy lifting. I think

Tim Kowal [52:05] The iron grip on it, yeah.

Jeffrey Lewis [52:07] I think that wraps it up. Unless there's any other questions you had, Tim.

Tim Kowal [52:11] No, that's it. Thank you, John, for for joining us. AI was used non specifically in the production of this podcast.

John Stobart [52:18] Yes.

Jeffrey Lewis [52:19] Nicely done.

Tim Kowal [52:21] All right, we'll see you next time.