Sacha Grief - State of AI - AUDIO EDIT
===

[00:00:00] 

Paul: Hi there, and welcome to PodRocket, a web development podcast brought to you by LogRocket. My name is Paul Mikulskis, and today I'm joined with Sascha Greif. He is a designer, developer, sole entrepreneur, and entrepreneur in general, here to talk about his most recent State of AI survey. Now, we've had you on before, Sascha, and I'm happy to have you back on again to talk about what you found out there in the world.

So thanks for coming.

Sacha: Thanks for having me. ~Um, ~yeah, like you said,~ I,~ I kinda do a bunch of things, but most of them center around running,~ uh,~ developer surveys. So the first one was the State of JS survey,~ um,~ 10 years ago, actually, first edition. And then every year since, I've done that survey again, and then over time, I've added the State of CSS survey.

And eventually, since last year, the State of AI ~or, ~or the full name is the State of Web Dev AI, because it turns out there's ~like ~15 other State of AI things. So I wasn't the first one ~to, ~to think of that. But,~ um,~ yeah,~ the,~ the idea is just to, ~you know, ~not only ask AI people what they're [00:01:00] using ~and, ~and what...

how they feel about AI, but more generally web developers,~ um,~ to hopefully capture both,~ like,~ pro-AI, anti-AI perspectives, and then everything in between and kinda get ~a, ~a representative sample of,~ uh,~ the state of things. But that's actually pretty hard to do because things are changing so fast in web development, but especially in regards to AI.

So I'm not even sure,~ uh, you know, ~how accurate those,~ uh,~ survey results are. But hopefully,~ uh,~ we can still track how the trends evolve from year to year and,~ uh,~ have interesting things to, to discuss. ~So, ~so that's basically,~ uh,~ where I'm at in a nutshell.

Paul: And for anybody listening,~ um, the, ~the state of AI that we're gonna be talking about today, it's not necessarily all hands on deck on the hype train. I think Sasha's here to bring us through some realistic check-ins with the community and,~ um,~ help us understand how people are using it, and like he said,~ most,~ most importantly, not using it or finding places where it's falling short.

~Um, ~and there's some interesting stats and numbers we're gonna get to talk about. But just to confirm, Sasha, here we're gonna be talking [00:02:00] about an agnostic look, right? Is that... Would you say that's the goal of the survey?

Sacha: Yeah,~ I,~ I would say that's definitely the goal, and the other factor is that I myself am not an AI power user by any means. ~Like, ~I have dipped my toe,~ uh,~ a little bit and I-- but ~I, ~I think my AI use is very unsophisticated still. So in some ways, maybe that's-- ~I mean, ~that's not really ~a, ~a guarantee of anything, but it does provide,~ like,~ an outsider perspective ~and, ~and I wouldn't...

~You know, ~it's hard to be objective about yourself, but I don't think anybody would say I'm, like, victim of the AI hype and then on the AI bandwagon. ~So, um, ~yeah, my goal is to, ~you know, ~try and identify AI,~ uh, you know, ~advantages and how people are making interesting use of AI, but at the same time also be realistic and,~ um,~ maybe even discuss some of ~the, ~the downsides ~and, ~and potential risks of AI, and hopefully that comes through in the survey.

~So, ~but you, you-- ~I mean, ~you can be the judge of that, ~I guess.~

Paul: ~Well, ~let me tell you, I'm selfishly [00:03:00] excited for this because you're talking to somebody on the hype train. You're talking to somebody that,~ uh,~ hosts podcasts, and I'd say 9 out of the 10 podcasts we do or talk about are AI adjacent or it has to do with what people are doing or s- or say they are doing with AI.

And I think this is gonna be a great grounding episode, not just for me, but for our general audience. ~Um, ~just to give some weight to the state of surveys that you run, how did you find the participants here? How many of them were,~ uh,~ responding? And do you know anything about, like, where they're coming from, experiences?

Is it mostly web devs? Talk to me a little about the audience, then we're gonna hop into some interesting stats.

Sacha: Oh yeah, sure. So this survey had over 7,000 respondents, which is pretty good for,~ uh,~ something that's, ~you know, kind of ~a niche topic i-in a sense, 'cause it-it's, ~you know, ~not web dev in general, but just AI. And most of the respondents came from our existing mailing list, so we have about 90,000 people on that,~ uh,~ accumulated over time from people [00:04:00] taking the State of JS, State of CSS, and all the other surveys.

So that's why I say it's the State of Web Dev AI, because we are asking web developers, right? ~So, um, ~pick a random example. If you see that one of the most used language is,~ uh,~ TypeScript and JavaScript and not Python, that's because we are asking specifically web developers those questions. If you looked at,~ uh,~ some other AI survey, maybe Python would be number one,~ um,~ and that might be because you are asking AI developers.

~Uh, you know, ~maybe I... Don't take my word for it, but ~you, ~you get the idea. ~Like, ~the... It's important to remember that the audience you select affects ~the, ~the data you get in return. And then,~ uh,~ you can actually see in the survey itself,~ um,~ the last section is the metadata section, which is where you can see all the information about the respondents,~ uh,~ besides the questions that they filled out.

~So, you know, um, ~what device they were using,~ uh,~ operating system,~ um,~ and including,~ uh,~ where... how they found the survey, which we aggregate through re-Referrer and,~ uh,~ also,~ like, uh, ~tracking codes and this kind of thing. ~Um, ~and so I think for this [00:05:00] survey,~ um,~ a big source was, like I said,~ our,~ our mailing list, and then also,~ uh,~ Theo,~ uh,~ the YouTuber,~ um,~ he shared the survey online on Twitter, so that sent us ~a, ~a nice boost of people, but,~ um,~ maybe around, I don't know, ten, fifteen percent, I think.

~Um, ~so the ma- the majority was still people who found the survey either through our mailing list or organically on social media. by the way, because ~we, ~we have ~a, a, ~a query builder tool, you could even filter out a specific subset of respondents if you're like,~ "Well,~ people who found on Twitter are gonna ~be, ~be like AI boosters, and I don't trust them."

So you can exclude them from the dataset and filter any chart,~ uh,~ like that.

Paul: ~Hmm. ~Okay. ~If, ~if you're a peruser and you really wanna nail down who's answering what. Okay, great. I di- I didn't know that about your state of surveys. Is that new? Is that n- new for this year, or

Sacha: ~Uh, ~I think

Paul: always had like

Sacha: three years that we've had that features, but we keep improving it and kinda tweaking the UI. And ~it, ~it's not super obvious, like it's a,~ um,~ small query builder link besides each chart. But yeah, because [00:06:00] it's more of a power user feature and a bit,~ uh,~ you need to invest a little bit of time to understand how it works.

I, I haven't made it super in your face 'cause I think most people aren't gonna use it anyway. But yeah, it's there ~if you, ~if you need it.

Paul: ~Well, that's, ~that's good on the audience. I hope that frames what we're talking about to some of our listeners. ~Um, ~o- on the results, Sacha, so we have interesting results to talk about. I have a couple numbers written down here on my Apple Notes on the side,~ uh,~ just so I don't forget and I can reference them right a- right away.

The first one that I wanna ask you about is, it's a number that shocked me, okay? ~It's, ~it's about the amount of people that are actually shipping AI-generated code. And so on your survey you said s- I'm r- and I'm reading here, "69% of developers are generating less than 25% of their code with AI." So my first question is, do we think that is an activation problem?

~You know, ~people just haven't converted to being AI native and so they're not shipping with AI. Or is it a,~ uh,~ metric that has to do with the actual efficacy and the actual,~ um,~ [00:07:00] usefulness of what people are producing with AI?

Sacha: So can you say that again?

Paul: Yeah. Do you think that the 69% of developers generating less than 25% of shipped code with AI, that, that metric right there,~ um,~ is it mo- Sorry, go ahead

Sacha: ~Uh, ~is that-- ~where do I, ~where do I say that? Because I think,~ uh,~ the figure is like fif- the average... Okay. We can cut out that, that part, but,~ uh,~ maybe

Paul: Elizabeth will help us and cut it out. We can waffle about, it's okay

Sacha: Maybe you can point me to the exact,~ uh,~ chart so I, I don't s- ~um~

Paul: ~Well, from, ~from your read,~ uh, maybe, ~maybe I wrote down the wrong one here, Sasha, in my notes. How many... What percent of developers are reporting they're shipping production code with AI that is ready to go?

Sacha: ~So, um, ~I think it's 20-- 29% of developers ship,~ uh,~ less-- use AI for less than 25% of their code. So you were saying like 69 or something? That seems very high.

Paul: it's 29. So 29% of developers are using

Sacha: Okay, that's-- Oh, is that what you said? Sorry, I misheard then. [00:08:00] Okay.

Paul: no, I said 69. I said it wrong. You were good to catch me, and thank you 'cause,~ like, we, ~we don't want the wrong thing ~in the, ~in the pod, so we'll cut that out. Okay. So

Sacha: start again.

Paul: let's start again. So one of the metrics is 29% of developers are shipping 25% or less of the code,~ uh,~ with AI, right?

Sacha: Yeah. ~Um, well, ~with the caveat that this kind of survey can't really tell us anything about ~the, ~the causation, right? It's only showing correlation, so it can't really tell us why people are doing this or that. ~Um, ~but I count myself in that group of,~ uh,~ developers shipping less than twenty-five percent of their code using,~ uh,~ AI tools.

And for me, at least, I can talk about my own experience, right? It's ~kind of ~a combination of different factors, maybe three factors. First of all, I have a lot of experience, and ~it's, ~it's not that I'm a better developer than other people, it's just that all this experience has taught me that you can waste a lot of time jumping on bandwagons too soon, right?

~Uh, ~and these days, I'd rather wait until other people figure out all the issues and edge cases, [00:09:00] and ~I, ~I'd rather be, like, a year late than a year early, I think. Which is not how I was, like, even ten years ago. Ten years ago, I was like, "Oh, I need to be the first so I can tell, ~you know, ~tell people about,~ um,~ my, my awesome,~ uh,~ experience being an early adopter," ~and, ~and so on.

~Um, ~but I would say the second factor is ~Well, ~there's a little bit of burnout sometimes. Like it took me a long time to adopt TypeScript, for example, whereas, ~you know, ~I have no doubt that TypeScript is ~a, ~a superior,~ uh,~ way to code JavaScript,~ uh,~ for a ton of reason. But it just took me a long time because, ~you know, ~one more thing to learn, changing your workflow, ~changing your, ~changing your tools.

~Um, ~yeah, just general burnout about ~the, ~the constant churn, ~I guess. ~And then finally, the last,~ uh,~ thing for me at least, is that I work-- I don't work on new projects very often. ~Um, ~I work on those surveys which have a big code base, which needs a lot of refactoring. There's a lot of dead code, there's a lot of things that kinda work this way because, ~you know, ~it was an edge case that I had to kinda quickly fix.

And so maybe I'm wrong on that, you can tell me, but my feeling was that AI tools were at [00:10:00] least maybe last time I checked, like a year ago, whatever, AI tools were very good at creating new things from scratch,~ uh,~ moving quickly to add code. But I was very worried that I didn't wanna, ~you know, ~add any more bad code to the existing,~ uh,~ huge amount of bad code that I've created,~ uh,~ already.

And,~ uh, I, ~I made a joke once that, ~you know, um, ~oh, today ~I, ~I spent so much time refactoring code created by ~a, ~a brainless entity that just produce, produces,~ uh,~ crap code to fix edge cases. And I'm not thinking about AI, I'm thinking about, talking about myself. ~So, um, ~yeah,~ I,~ I was skeptical of AI for that reason.

~Like, ~would it actually hurt,~ uh,~ maintain-maintainability? And would it be able to kinda understand the scope of a pretty large existing project? So yeah, I'm curious to know how you come,~ uh,~ at the-- these issues. ~Like ~is-- are those things that you've felt as well,~ uh,~ maybe?

Paul: Oh, definitely. And as somebody who's very AI forward in the way that I develop apps these days, it's, ~um... ~You don't start to control this [00:11:00] until you're really at a high level of being okay with token spend and focusing that token spend on one project. ~Like, ~the $200 plan from Claude doesn't go that far,~ um,~ once you really start to build in the automated systems.

When people are like, "Oh, I have agents running for days," you can bet your ass they're spending a lot of tokens on the system that lets it do so responsibly. And then there's a trade-off question, like what's the opportunity cost of me just developing? ~Like, ~could I do it faster? ~Um, and, ~and ~that, ~that begs the question, Sasha, right?

It begs the question of, is this a conversion problem? Is it a problem? Is it a problem with the underlying technology, which is AI? Why are such a low percentage of developers shifting shipping such a low percentage of code using these tools? And ~so, like, ~if you were to take a 10,000 step foot,~ uh,~ 10,000 foot view,~ right,~ in the sky ~at, ~at the whole survey, do you think that it is everybody's trying AI and they're ~sort of ~feeling a little iffy about it?

Or do you think this is,~ like,~ a fundamental [00:12:00] thing that isn't being talked about because of the hype train that you're seeing bubble through in the state of AI survey, and it's actually a problem with the tools? It's a problem with what we think they're capable of

Sacha: ~Um, I, ~I think it's interesting

Paul: From the answers. ~From, ~from the answers, not... Yeah

Sacha: my, my read on the situation is like: "Oh, wow, I can't believe so many people have adopted AI," and you're like, " How comes," ~you know, ~"how come there are still some people who haven't been converted to the holy word of,~ uh,~ AI yet?" So it's kinda two,~ uh,~ different perspectives.

But,~ uh,~ I think,~ like,~ first of all, speaking for myself again,~ uh,~ and s- that's something that's also,~ uh,~ might show through in the survey this year and probably even more next year. I'm kinda worried about,~ uh,~ the cost of AI now because I feel like the honeymoon period where AI was subsidized by,~ uh, the, ~the nice venture capitalists in Silicon Valley is coming to an end, and now we're gonna have to actually pay for,~ uh, what we, ~what we consume.

And as a solo entrepreneur, I find that very scary because if you have ~like ~a, ~you know, ~a $50,000 bill that's coming out of your VC money, that's one thing. But if it's coming out [00:13:00] your own bank account which you use to feed your family, that's, ~uh I, ~I'll just have to change my identity and move to a different country at that point. ~Um, ~so that's one factor that I think is gonna hurt adoption,~ um,~ or maybe drive people to self-host AI more or, like you said, become more,~ uh,~ deliberate about how they use AI. And then the other thing, yeah,~ I,~ I don't know. Maybe some people are not that interested in using AI. I personally think,~ um,~ we should encourage that because, ~you know, ~it's like something like woodworking.

Some people use power tools, some people use hand tools. And the people who use hand tools, not only can they do things,~ uh,~ that you can't really do with power tools, and they can also be more efficient, and also they're preserving knowledge that's,~ uh,~ very valuable. And I see the same thing with,~ uh,~ programming, right?

If you don't use AI, you're gonna maybe come up with solutions that are un- that are more unorthodox that AI wouldn't think of, and of course, the other way around as well. ~Like, ~AI will definitely come up with,~ uh,~ solutions that humans [00:14:00] wouldn't,~ uh,~ come up with. But all the same, I think i- if I were running a company, and if I had a- ~um, ~employees that do not use AI,~ uh,~ not only would I not,~ like,~ penalize them ~or, ~or anything, I would treasure them.

~Like, ~I would be like: "Okay, you guys keep doing your thing. We need people like you." Because, ~you know, ~the next generation of AI needs to be trained on something, right? So if it's all,~ uh,~ AI feeding AI,~ um,~ will it really work out as well as it has up to now? I'm not sure about that. So sorry, I know you, you asked me to answer more on ~a, ~a database perspective, and I'm giving more of my own,~ uh,~

Paul: No,~ you're,~ you're totally fine. The goal of this session is to get your raw, unfiltered thoughts because, look, we're listening to the guy ~who sees, ~who sees a lot. You see a lot. You see a lot what people answer, and it's hard to,~ like,~ take the visceral, guttural read about,~ like,~ what does this... Oh, hi, Elizabeth.

Oh, you're all good You're... Okay, I'm gonna rewind and say what we want, Sasha, is your unfiltered read. And so it's okay that you're talking about databases or whatever because y- you see everything, you see what these people are answering, [00:15:00] and the visceral, guttural feeling about what do these responses mean ~is, ~is really what we wanna get at.

And ~so, like, ~actually ~not, ~not looking at another stat or percentage or anything, just looking at the responses in general, do you think that people are more excited or more scared about AI? Because in the response, it's private, right? In the responses that people give. So people would be inclined to be more honest than,~ like,~ on the internet.

'Cause ~if, ~if you're on the internet and you say, "Oh, no AI, it's not good," who are you? Are you somebody that doesn't want the new technology? So I'm interested to see in this more truthful space, do you think that people are more scared,~ um,~ or more excited in general about,~ uh,~ the AI in their d- in their workflows?

Does that 25% directly correlate to how they actually feel?

Sacha: ~Well, ~I think that's what's so interesting about AI and ~kind of, mm, ~takes it like,~ um,~ i- makes it different from any other technology what-- that we have previously, because I've experienced a lot of,~ uh,~ hype trains, ~you know, ~whatever the new JavaScript framework, the new this, the ~new, ~new that. And usually, the people who use it the most are the most positive, [00:16:00] and to the point where people...

Y-you can really, ~you know, ~it's a bit, ~you know, caricatural, ~caricatural in a way because obviously every technology has downsides, but then people are willfully blind to the fact that, oh,~ well,~ maybe, ~you know, ~React will eventually develop some downsides that, ~you know, ~might want to make you use something else.

And no, React is the best, and I love React, but, ~you know, ~even I can admit that it has some sh-shortcomings in some areas. And in areas,~ uh, uh, ~sorry, AI, I feel is unique in that people, even people who use it do have a more,~ um,~ balanced view of it, and ~they do, ~they do feel scared a bit. So if you look at things like,~ um,~ a question,~ uh,~ we asked about do people,~ uh,~ fear for their job safety due to AI?

Most people do,~ uh,~ whether they use AI or not. And even if they,~ uh,~ generate most of their code with AI, and maybe even, ~you know, ~because they do,~ uh,~ they're realizing that AI can be a threat to ~their, ~their job. And they're also maybe realizing that AI can pose some,~ uh,~ let's say, enviro-environmental concerns or,~ uh,~ there's a whole,~ uh, uh, ~list of other risks associated with AI.

~And, ~and I think it's [00:17:00] interesting because like you said,~ uh,~ sometimes you can have this view of,~ um,~ AI developers being very,~ uh,~ hype-driven and seeing everything with,~ uh,~ AI-tinted glasses. But in the survey at least, I think people-- I was surprised by how,~ uh,~ balanced people were in ~their, ~their assessment of AI independently of whether they use it or not, basically.

Paul: One thing in your survey is most people said they do get a productivity boost, right? That was a unanimous majority, right?

Sacha: Yeah. So again, I think the drives are the same thing where AI does offer a productivity boost, and fighting AI in a way on that ground, I think is a losing battle. ~Uh, ~but bec- just because you are,~ uh,~ aware of the productivity boosts you can get from AI a- and the very real advantages you get from using AI, doesn't mean that you can't also,~ uh,~ be conscious of the downsides.

And I think that's, for me at least, that's ~kind of ~the path I wanna adopt, and I f- the path I wish most people would,~ uh,~ adopt. Because like you said, I don't know, I feel [00:18:00] the discourse right now is very polarized, and people who are, quote-unquote, "anti-AI" kinda get dismissed right away because they,~ um,~ maybe haven't used AI or kinda are reject- rejecting it,~ uh,~ in block, like outright.

And then the people who are pro-AI are kinda, ~you know, ~putting their ~head in, ~head in the sand and kinda ignoring,~ uh,~ the cost increase and,~ uh,~ job loss and all that. And I think we need to grow past that phase of the discourse and kinda look at both sides, yeah,~ uh, in a, ~in a way that makes sense.

Paul: ~on, ~on the,~ like,~ people who haven't used it or people who might not really have given it ~a, ~a truthful try for their workflow, I'm curious,~ like,~ what questions or answers might have revealed something about this. ~And, ~and ~so, like, ~one example is ~my, ~my younger brother recently entered the workforce, okay?

And at first he says, ~you know, ~"AI..." He loves programming. He loves programming by hand. ~He, ~he went to school, right? So you graduate school, you're excited, you wanna show what you are. And then he goes, "The AI, it makes so many [00:19:00] comments," okay? ~"I, ~I have so many comments in my file. This is ridiculous. I don't know how this could ever make a good comment that I feel ~like ~helps me."

And in my opinion, that was a skill issue, right? It is a, ~uh... ~It is an issue of maybe you didn't ask it correctly, maybe you didn't have the right rules. And I'm wondering,~ like,~ in the survey, what sort of,~ uh,~ responses might have revealed either people complaining about something that wasn't really an issue, or maybe they are complaining about something that shows ~a, ~a bigger takeaway, like ~in the, ~in the way that we're adopting this technology, or in the way that we're viewing everything like a nail, okay?

~And the AI ~and the AI's a hammer, so we just put it on everything. Maybe that's the problem. We-- It does-- AI shouldn't be in every single building space. So I'm wondering how much ~of the, ~of the responses were like, "Oh, bad variable names, too many comments," ~uh, ~"Bad structure," which are fixable, versus things that might have actually degraded a code base and [00:20:00] you...

there's nothing we could do about it. Skill issue versus AI issue. What questions might have,~ like,~ revealed some of this?

Sacha: Yeah, that's interesting. So in ~the, ~the risks and pain points section,~ um,~ we have a couple questions that cover that. So we ask about pain points,~ uh,~ and halu-hallucina- hallucinations and inaccuracies is number one, which I think is not a skill issue. ~Like ~that's something that LLMs are pretty notorious for.

~Um, ~and then code quality is number two, but it has gone down from,~ um, uh, ~sixty-three percent of people,~ uh,~ who cited code quality to fifty-three percent. So I think things are getting better. ~Uh, ~prompt comprehension has gone down,~ uh,~ seventeen percent, so it was,~ uh,~ a much bigger concern last year compared to this year.

So I think we are seeing that,~ uh,~ as LLMs improve, that's reflected in the issues that people struggle with, right? ~Uh, ~we also have a question about missing features,~ um,~ which does not have,~ um,~ have a,~ um,~ comparison with last year, but the first one is,~ uh,~ truthfulness, which echoes the problem with,~ um,~ hal-hallucinations and then [00:21:00] long-term memory, up-to-date knowledge, accountability.

So over time, I wouldn't say it's-- Like I don't see skill issues,~ uh,~ being much of a factor here. I think it's mostly,~ um,~ LLM's own,~ uh,~ limitations. Yeah. But also, ~I mean, ~speaking of skill issue, like there's-- I have a skill issues. Like I said, I'm not an AI power user, so I think if I were,~ uh, there would, ~there would probably be some questions, some better questions I could have asked to try and identify how people are actually using AI ~and, ~and not just like what they are using in terms of models or how they feel about it.

But I think some people have pointed out that the survey does lack a little bit in terms of,~ uh, you know, ~discerning that more granular view of how people are actually using on a-- AI on a day-to-day basis, and hopefully I can,~ uh,~ fix that,~ uh,~ for next year.

Paul: Yeah. The quality of the question and the actual niche or takeaway that it's trying to extract ~in, ~in the way it's asked is really important. And I'm sure there's like ~a, ~a question or two that you thought you were asking something, but the responses come back and [00:22:00] you're just like, "Oh my gosh, like I did not expect it to be so high or so low."

What's one question on the State of AI survey that did this for you?

Sacha: ~Well, ~so I have an example of a question that I kind of-- two questions that I messed up. So I asked,~ uh,~ last year I asked people how much they spend on AI personally, and then how much their company spends on AI. Because at the time, even last year, ~you know, ~when you were talking about AI spend, it was like, okay, do you sign up for ChatGPT or not?

And ~do you, ~do you pay whatever it was, $30 a month? And the company might pay a couple hundred dollars. And so this year I asked the same questions phrased the same way, and I realized,~ like,~ th- it didn't make sense anymore because first of all, like company spend, AI spend, you can't even know how much it is anymore.

~Uh, ~you can't even measure it sometimes if you're,~ uh,~ an employee. So at most you might know how much,~ uh,~ your company spends on you, like on a personal basis. ~Uh, ~but you can't really know how much the company spends in total. So I actually just removed that question altogether because,~ uh,~ people weren't [00:23:00] able to answer it accurately.

And then even the other question, how much do you spend per-personally on AI? ~Uh, ~I think I kept it, ~um~

Paul: Oh, you capped it. Smart

Sacha: ~Well, ~because it was a multiple choice,~ uh,~ question, so it goes up

Paul: Oh, okay

Sacha: So the highest bracket was,~ uh,~ between ~a, ~a thousand and five thousand,~ um, uh, ~US dollars monthly, which is a lot, but we know now ~that ~that's it could, can go ~much, ~much higher than that. So I definitely nee- definitely need to change the brackets for,~ uh,~ next year and also kinda,~ uh,~ make it more accurate ~whether I, ~whether I'm asking about,~ uh,~ personal spend on a personal level or personal spend in terms of what your company is spending on you.

~So, uh, ~those two questions are examples ~or, ~or things where, you know, you might think-- You might ask the question one way, and then the data shows you that you kinda missed the mark,~ um,~ because of the way the question was phrased, basically. And because things move pretty fast, so the way you ask a question one year might not work for the next.

Paul: What about,~ um,~ what people are mostly complaining about ~or, or, ~or grieving [00:24:00] about ~the, ~the shortcomings of AI? ~Like, ~obviously ~the, the, ~the number one thing will be quality. The thing you give me was not good, or the thing you give me was disorganized and I can't use it. ~Um, ~I'm curious if you had any responses about how it's affecting the people themselves, how it's affecting ~or, ~or the way the AI works itself.

Maybe people have specific problems or issues they have with ~the, ~the way the models talk to them,~ um,~ separate than the outputs, 'cause we all know the outputs are questionable sometimes. So other than that, what are people grieving about when it comes to embedding AI into their daily lives as developers?

Sacha: ~So, ~so there is this,~ uh,~ chart about,~ um,~ AI pain points. So li-- we mentioned,~ uh,~ hallucinations already. ~Uh, ~number four was,~ uh,~ financial costs, and what's interesting is that has gone up a lot. So last year it wasn't-- it came in number seven, I think. And this year it's number four, and it gained,~ uh,~ eleven percentage points.

So I, I find it interesting that,~ uh,~ financial cost was ~kind of ~a non-issue, and now it's becoming much more [00:25:00] prevalent as I think companies are realizing that they need to monetize ~and, ~and if they wanna, ~you know, ~IPO or whatever, they need to show good financials ~and, and, ~and all that aspect. ~So, um, ~yeah, I think it's gonna be-be-- become a big factor.

~Like, ~just the last couple weeks, I keep seeing threads about people, ~you know, uh, ~posting screenshots of huge AI bills. ~Like, ~"Whoops, we have a hundred thousand Anthropic bill this month." ~Um, ~these kind of things. And I never used to see that until a couple months ago. So I think ~that's, ~that's a big factor.

And then in terms of more of,~ uh, the, ~the psychological aspect,~ um,~ we talked about,~ uh,~ job security, so people are worried about that. And I haven't-- ~Uh, ~so this survey didn't have that question, but I will ask it in a future survey, which is,~ um,~ how-- ~what's, ~what's AI's impact on your mental health? ~Like, ~is it,~ uh,~ helping your mental health or not?

Because I get the feeling that even if you do use AI and it is making you more productive,~ uh,~ there's a way that AI could also be more stressful because maybe you're trading,~ uh,~ just coding things by yourself, which is kinda [00:26:00] low stress,~ uh,~ to, to-- for, ~you know, ~now having to review AI code and you may be worried about bugs or security issues and maybe overspending.

And ~so, ~yeah, I think it's gonna be interesting to see,~ like,~ if we get to a world where even though the code quality is perfect,~ uh,~ it turns out that we actually don't enjoy u-using AI so much because of all these other factors that people maybe hadn't really,~ uh,~ taken into account before.

Paul: ~Right. ~It almost makes me wonder, is productivity good for your mental health? Probably. Probably, but the right types of productivity.

Sacha: ~Hmm. Right.~

Yeah

~Uh, ~i-if you're just ~measuring, ~measuring lines of code, then maybe not. ~Like, ~maybe now you have ~a, a, ~a 50,000 line of ~code ~code base instead of a 5,000 one, and that's not gonna be good for your,~ uh,~ mental health going forward ~if, ~if both code bases do basically the same thing, right? Or maybe, ~you know, ~I think,~ uh,~ there's also a good point to be made that we're gonna end up with code bases that do too much, right?

'Cause it's becoming so easy to add features that,~ well,~ now your,~ uh,~ your whatever,~ uh,~ email client also has,~ um,~ [00:27:00] an image editor built in for whatever reason 'cause someone said,~ "Well,~ why couldn't people, ~you know, ~edit attachments once they upload them? ~Uh, ~that'd be cool." And then now y-with AI you can do that, but now you have to maintain that code.

And of course, AI is gonna maintain the code as well, but at some point, maybe... ~You know, ~I like to,~ um,~ compare AI to a living organism, and ~I, ~I'm sure it's not an original thought, but,~ uh,~ or living organisms sadly have to deal with cancer. And I know it's not ~a, ~a very,~ uh,~ nice metaphor or nice thought, but at some point, do we ~end, ~end up with this kind of,~ uh,~ thing where AI apps just gonna grow exponentially and kinda die under their own weight because of that?

~Um, ~I think that's a very real possibility that people are probably not fully,~ uh,~ considering right now. So yeah, I don't know what you make of that, but,~ um,~ I think there's gonna be a lot of other risks and issues that we need to,~ uh,~ grapple with going forward.

Paul: That's, it's, that's a very real effect. I actually have a story. So I made this,~ like,~ little site for a friend of mine [00:28:00] called cardcanner.com. So you could scan a business card, and it sends you a contact back on your phone. ~Well, ~you text it. You text the business card, and it sends you back the contact. Just a free app,~ like,~ to help him snap business cards.

And he goes, "Oh, could this import to Salesforce?" And then somebody else uses it, "Oh, could th- could this go to Google Sheets?" And somebody else comes and say, "Oh,~ could,~ could this have,~ like,~ a email button right in there?" And eventually ~I, ~I realized, I was just like, "Oh, my gosh, there's no... I'm not interested enough in this app, and it's gonna be a whole maintenance burden."

That's not what... This was just meant to,~ like,~ take a picture of a business card and send me a contact card back, and now it's becoming this,~ like,~ amalgamation of other integrations that I have no interest in maintaining anymore, and wh- which kinda sucks. ~Um, ~I could totally see that being the s- ~the fattening, ~the fattening and slow death ~of, ~of some great small services out there

Sacha: ~I mean, ~people do talk about,~ uh,~ enshittification, right?

Paul: Oh yeah

Sacha: ~um, ~something that's,~ uh,~ very real, and there's a way in which AI can help fight that by,~ um,~ making products better, but also [00:29:00] a way in which it just makes it even worse by giving us bloated products that kinda do everything,~ like,~ kinda badly and in a buggy way.

And yeah, so I, I hope it doesn't come to that, but at least ~the, ~the survey definitely wants to capture that angle of things as well and not just ~the, ~the cool part. ~Like, ~"Oh, look what I built. Look what I vibe coded,~ uh,~ with,~ uh,~ my, my shiny new AI,~ um,~ agent," and so on.

Paul: Are, is there anything on the survey that you think revealed anything about the way interpersonal communication is working within organizations?

Sacha: ~Um, ~not really. That's interesting. ~Um ~Yeah, ~I mean, ~I didn't even consider that. ~Like, ~do you mean in, in the sense that people might rely on AI to answer emails and talk to each other

Paul: Sure. Yeah, that or, ~you know, ~people are finding reduced communication barriers, heightened

Sacha: う

Paul: communication barriers. ~Um, ~like I heard of this one, and I can't remember the company, but there's this one big company, I think it was on the Fortune 500, where the CEO was saying, ~you know, ~"When you email me," and the CTO, "When you email [00:30:00] me, my agent will respond, and then the e- agent will triage to me."

I'm like, w- this is an additional communication barrier. Like I, if I have the authority to email the CEO, and at the human level I know it's all good, I should just be able to email the ce- the CEO. ~I, ~I don't want to talk to some agent. I'm curious like how organizations are evolving in this way. ~Um, ~but if it's mostly like on, on web development ~and, ~and how people are, yeah, using it themselves, maybe not

Sacha: ~I mean, ~it might be a good question. So the next survey, the... I don't know when this is coming out, but the currently running survey is the State of CSS, which is not about AI, although it does include the same question that we had here about,~ uh,~ what percentage of code is AI-generated. But in this case, it's specifically what percentage of your CSS code is AI-generated.

And I can tell you from a prelim-preliminary results at least, the percentage is way lower, like the chart is almost flipped. ~So, um, ~I don't know if it's something specific to CSS or,~ uh,~ respondents,~ um,~ but at least [00:31:00] our,~ uh,~ cohort is using AI way less. So that's already interesting. And then the next survey after that is gonna be the State of Devs, which is about all the aspects of a developer's life that are not code, that are not programming.

~Um, ~so last year it was,~ uh,~ workplace, health, career, things like that. But this year it is gonna include ~a, ~a dedicated AI section because I think, ~you know, ~you can't-- just can't avoid it.

Paul: ~Right~

Sacha: ~uh, ~this question about,~ uh,~ yeah, interpersonal communications, maybe that's something I can add in there to try and learn more about that.

That's,~ uh,~ interesting,~ uh,~ an interesting topic.

Paul: What is,~ uh,~ a question that you think you would definitely change for next year? I know you mentioned the cost one, which I, I-- you know, that totally makes sense as the pricing models have literally changed quarter to quarter. ~Um, ~but in terms of how people are using it in their workflows, what would you wanna change for the next State of AI survey given what you've learned this year?

Sacha: I wanna do a better job capturing more-- in a more [00:32:00] concrete way how people are using AI. ~Um, like, ~cause, ~you know, uh, ~I think I have a, ~kind of ~a basic, very basic view of AI where it's like, for me, it's still,~ uh,~ ChatGPT, and you type something in, and ChatGPT answers. And ~like, ~of course, I know there's more to it than that, but I think the survey is still stuck in that era a little bit.

So next year, I wanna properly reflect like how people are controlling their agents and like what, ~you know, ~approach they have to keep costs down and all. 'Cause ~it, ~it's gone from just, ~you know, ~prompt engineering to more something like there's a lot more factors than just typing in the right words, right?

~So, uh, ~I don't think the survey captures that currently, and ~I, ~I wanna improve that for sure. ~Um, ~but I don't know if you have any suggestions about that because you're definitely ~the, ~the expert,~ uh,~ on that at least.

Paul: Yeah. ~I mean, I, I, ~I feel like getting at people's,~ uh,~ issue tracking and state tracking and context management systems and what that means for each person, 'cause it means something different for everybody,~ um,~ would be very interesting. 'Cause there's some people that evidently just figure it out, and they have something that works for them.

~Like, ~there's some [00:33:00] people you see online, you're like,~ "Are, are they, ~are they bullshitting me or are they really running an agent for three days straight doing real work that they keep?" And getting at,~ uh,~ how people are managing that context and managing those state tracking or issue tracking systems, or how you even ask that to extract it correctly.

~You know, ~I listened to a podcast from AI engineer or,~ uh,~ a talk from AI engineer, and they said, ~you know, ~"GitHub issues, it's good for humans. It's not made for tracking on AI, and we have not found success there," which really surprised me 'cause I would think, "Oh, GitHub issues, it's like that's the spot. I...

You just need one tool, and it should be great." But apparently it's not, and people are,~ uh,~ finding a sharper edge,~ uh,~ m- rolling their own state tracking and issue tracking.

Sacha: ~Well, ~I have a theory that the endpoint for a lot of that is basically self-evolving code bases or self-evolving apps. And what that means is basically the app would,~ uh,~ find its own issues and errors and bugs, which you can already do with a couple error tracking [00:34:00] systems. ~Uh, ~the AI can also write its own pull request, and then another AI can review the pull request, approve it, and deploy it.

So essentially,~ uh,~ apps develop their own immune systems,~ uh,~ to fight their own bugs, just like we do. So that's why I was talking about,~ uh, uh, ~the organic aspect of AI. So that's my own,~ like, uh, ~pet theory. ~Um, ~not just my own, of course, but that's something I've,~ uh,~ been thinking about, and I'm not sure how I would measure that in a survey, but I think it would be interesting to know how close we're getting to that,~ um, if, ~if at all.

And so maybe ~I'll, ~I'll add something along those lines,~ uh,~ next year

Paul: ~Uh, well ~taking a s- a summary or a step back, Sasha, ~like ~is there any advice, and I know you might say,~ "I,~ I can't give advice. ~I'm not, ~I'm not this AI guy." But like you see all the responses and how they change year to year, and ~you see ~other state of surveys of how people are looking at the changing landscape.

So do you have any unsolicited advice to kinda give listeners about like your third-party perspective about the hype and what people are investing time in [00:35:00] that might scare you or surprise you that you're like, "Hey, maybe you should think about this again"?

Sacha: Yeah, ~I mean, ~I would say don't dismiss AI. That's, but that's pretty obvious. Even as someone who doesn't use it that much,~ like, you know, ~I'm just trying to be realistic. I don't think it's gonna go away, even if ~the, ~the ~so-called ~AI bubble pops, even if costs go up. I don't think we're ever gonna go back to our previous way of working.

I think ~that ~that ship has sailed,~ so, um... ~And then on the other hand, if you-- ~I, ~I also think you don't need to feel too much FOMO, even if you are not a power user like myself, because I think ~the, ~the one thing AI is good at i-is helping you catch up. ~Um, ~and th-there's a learning curve, but I think ~it, ~it's by the nature of AI being able to, ~you know, ~talk to you,~ it's,~ it's smaller than with other technologies.

Let me know if I'm wrong on that but, ~you know, ~when you have to learn TypeScript, if you wanna get up to speed, you can't really compress that time. You have to go through the documentation, and AI kinda lets you cheat a little bit and go at your own pace a ~little bit, ~little bit more. So that's why I'm not too worried, at least speaking personally.

I kinda feel like whenever I finally decide to, ~you know, ~make that jump, if I [00:36:00] do, I think I can catch up because things will have improved anyway ~and, ~and,~ uh,~ AI can tell me what to do, how to best use my time. And ~so, ~yeah,~ the,~ the third thing would be, let's,~ uh,~ keep in mind the issues with AI, because I think it's kinda easy to sweep them under the rug.

And for me,~ the,~ the most important is,~ uh, well, ~global warming. We haven't really touched on that before, but that's the most important issue facing humanity today. AI requires a lot of energy, and even though, ~you know, ~one,~ uh,~ Google search,~ um,~ might not take up that much energy, even with AI features enabled, if you multiply that by the number of Google searches that happen every day, to me, it doesn't seem like the best use of our limited, ~you know, ~foss-fossil fuel supply that we have currently and maybe,~ uh,~ limited amount of energy that we have to kinda enable AI ~everywhere, ~everywhere and Of course, sure, if we can use AI for,~ uh,~ vaccine research or whatever, let's do that.

Let's spend all the energy and money we can on that. But right now, it [00:37:00] seems to me that a lot of AI money is being spent a bit,~ uh,~ in, in-- on things that are not needed. And so I, I have no illusion that whatever I say has any impact whatever on anybody. But yeah,~ I,~ I wish the tech community as a whole would be ~able, ~able to kinda keep both things in mind at the same time, like the very real,~ uh,~ advancements in AI that we're making and the productivity gains.

But also, maybe we don't need to have all the productivity right now and have all the widgets in our apps right now if that comes at the cost of more important things. I don't even re-even remember what your question was, but I just went on a rant. Sorry.

Paul: It was ~the, ~the advice looking at all the state of surveys and looking at

Sacha: Yeah, my advice is

Paul: you would give

Sacha: if you have a chance to,~ um, you know, ~publicly,~ uh,~ be vocal about those things, do ~so, ~because otherwise it gives the impression that everybody is just full on AI pilled and doesn't even realize the downsides. So yeah, don't be shy about,~ um,~ making it known that you are aware,~ uh,~ sorry, aware of the downsides without going full-blown [00:38:00] like AI doomer or anti-AI and pretending that AI doesn't have any advantages.

Find a sweet spot, that would be my advice

Paul: Awesome, Sasha. ~Well, ~if people wanted to look at the survey, participate in your upcoming surveys, where would they go?

Sacha: So you can go to stateofai.dev, and you can,~ uh,~ sign up for our mailing list here, and you'll know about the AI surveys and also all the other surveys that,~ uh,~ we do. And right now, you can also go to stateofcss.com to fill out the State of CSS survey and probably in a few days or weeks stateofdevs.com to fill out the State of Devs survey

Paul: Awesome. ~Um, well, ~Sasha, I appreciate the grounding of the realistic conversation here, which is what we're using the technology for and being vocal about its actual cost on humanity, 'cause there's a lot of ways people are using it,~ um,~ that are questionable. It's almost like saying, "Oh, we invented the car, and now I want everybody to drive people places they need to go," which is [00:39:00] not what happened with cars.

Sacha: That's a perfect analogy. Like cars, nobody's saying-- would say like cars aren't useful. But at the same time, it's very obvious that cars, if you design cities around cars, for example, that has downsides. And if you-- maybe public transportation can be better than cars for,~ uh,~ the environment and other things.

Maybe not every single person needs their own cars or multiple cars. For me, cars and AI ~are, ~are like almost they map one-to-one because, yeah,~ um, uh, ~I'm just always shocked that, ~you know, ~we seem to never learn the lessons of the past and we're like,~ "Well,~ yeah, cars, we ~kind of ~went overboard, but AI, no.

Yeah, let's go full steam ahead on that."

Paul: It's a different thing.

Sacha: Yeah,

Paul: Yeah.

Sacha: drive,~ uh,~ carefully.

Paul: usually I end with the where can we find the thing and where can we find you? But I wanted to end with this 'cause I do think it's a really important dialogue to close out with. ~Um, ~and I do appreciate you bringing us back to that third point of realism a- at the end.

~So, um, ~one last thing before we [00:40:00] close out. If people wanted to listen to you, Sasha, more, where do you post online?

Sacha: ~Uh, ~bluesky,~ um,~ @sashagrieve.com,~ um,~ or the surveys. But yeah, if you wanna listen to me, you probably shouldn't because everything I know is in the surveys, and everything I say beyond that is just my own,~ uh,~ subjective point of view. ~So, um, ~always rely on the data if you can

Paul: ~Well, Sasha, ~Sasha, excuse me, Sasha, thank you so much for your time,~ uh,~ and coming on once again and talking about the 10,000-foot view of what you're seeing in your amazing state of surveys, and look forward to having you again in the future for the next one