episode23

00:00:00 Speaker: Welcome back everyone to the VS Code Insiders podcast. The behind the scenes with the team that builds your favorite code editor in the entire world. I'm your host, James Montemagno, and with me, the one and only Connor. Pete. How's it going? Connor. Yeah, it's going pretty well. Um, yeah, I'm excited to be on and talk about the cool stuff that we've been working on for the past few months. Yeah, it's always an exciting time in the world of VS code, and especially everything that's happening with agents and sessions. And that's what I want to talk about a little bit today, specifically around agent host protocol. But before we do that now, people may have seen you on like, you know, some of the agents day and things like that on the VS code channel. But we want to introduce yourself to all of our listeners. Yeah. So my name is Connor. I've actually been on this team for about seven years now. So I'm, I'm kind of an old timer, but, um, um, I kind of like worked in like debugging, testing. You might have seen me a few years ago talking about that stuff on our live streams. Uh, but recently, of course, I've been, uh, working more in the space, doing more. Um, you spend much time doing stuff here. So if you've checked out the cool features that VSCode has, um, uh, and then, uh, more recently this year, I've been, uh, helping our efforts towards the agent host protocol, which, um, yeah, as a whole set of. Yeah. Anyway, sorry. Uh, yeah, yeah, it's a whole bunch of good stuff, which I'm excited to dive into, uh, because that entire space is evolving and it's fascinating. You're talking about MCP. I was just playing around with the, the GitHub MCP, and they have a bunch of MCP apps built in now. So when you go to create an issue, you get that beautiful UI. Um, I'm a huge MCP fan and I know it continues to evolve and they're working on new specs and things like that. Um, so yeah, how was it working on MCP as it was evolving since it was changing so fast? Yeah, it was really cool. Um, actually I first started working on MCP in VS code, I think in like March of last year. So as well, like MCP was, was still like early on, I think had been around like six months at that point. Uh, but even like, at that point, I think a lot of clients and the client is like, like, like this code, like a lot of clients really hadn't like exercised the full power of MCP. And therefore, you know, a lot of servers hadn't really used all the advanced features yet. Um, and so a lot of it was kind of figuring out, you know, like, how do we express all these things that we can do in theory? Like, how do they kind of map onto the like UI application flows that people expect? Um, and like all of that, like was like learning both for us as well as I think learning for, for people like building, um, MCP as well. Because like some things work, some things didn't like in the most recent version, there have been like a few things that have been deprecated that just were things that never really found their place. Uh, but there's also been like a lot of really cool new things like MCP apps, as you mentioned that people show like really nice UI really have, really have really rich interactions, um, like inside of their clients, which is super cool. Oh that's awesome. Yeah. I'm a big GUI person, hence I gravitate towards VS code and Visual Studio and anything that's visual. So I was really excited to see MCP apps come out. And for me nowadays, and as we're talking to a lot of developers, the things that we're actually working a lot with are these agents and specifically across sessions. And when the agents window came out, it wasn't just that we were working with one set of agents. All of a sudden we were working with a cornucopia, a plethora of agents everywhere around us, across multiple repos. And what I wanted to do today is like you were talking about the agent host protocol. But I think even before we go into that again, I think it's worth revisiting if people are opening up vs code. I guess this is the end of June twenty twenty six. Like what is the current implementation? I know we have like local copilot CLI, we have cloud, but like, how does that session stuff work before we even get to how it's evolving? Because I think it's good to know. Yeah. So actually it's actually changing like, like as we speak. And actually, I think by the time this podcast is out, we actually might have changed over to the agent host version by default. So it's very much kind of like an evolving area. And the whole team is kind of like working on this, squirming around and trying to make sure that it's up to par. Um, and even better than what came before. Um, and so I can like, uh, and so if, if listeners, like, if listeners haven't really heard of like all these different ways, different agents, um, I'll probably go into that a little bit. So, so if you use like code, I'll traditionally like we had, uh, like what we called like local, um, like agent experience, essentially like the local agent loop. And that was essentially running inside of like the process, the extensions for vs code window. Um, and so you like ran it there. And then like when you close the window, it went away. And then if you wanted to run it in a new workspace, you had to like open a window in that workspace and then operate there. Um, and I think that was like fine for a while. Like, I think it was fine. Maybe like pre, maybe like pre opus, I would say, because, um, you know, before that point, you really had to like monitor your, your agents closely. Like even, you know, the best models like sounded like they were really good, but, but they still couldn't really like go off, go off on their own for more than like a minute or two without having, having like used like step in and, and kind of guide them. Um, but that changed this year. Um, and so, um, actually, even before that, we had, we had support for, uh, Brooke trees in, in vs code. So like if you're doing a new agent session, you could like spawn that inside of a tree. Um, and this kind of worked, but there were like some kind of weird things just with, you know, you're in a tree, which is like, um, it's basically like a detached, like separate, like git clone that's attached to your main, like git tree or you're mean, like get, get workspace. Uh, um, and this kind of worked like it was good enough for people who like used birch trees to really do those kind of parallel tasks. Um, but it wasn't really all the way there. Um, and that's just like architecturally, there's just lots of things that kind of weird with that because vs code was kind of built in like one window, like one folder, a kind of world. Um, and so, uh, something started changing actually, like early this year was, was, might have played with the, the vs code agent window. Um, and um, that, that basically is more of a, a, a high level home where, where all your agent sessions can live. So, uh, I can have a, like a yes. You can have like, you know, um, like a chat box. You pick a folder like where the agent like works in and then, um, you can spawn it off and then you can like mount again, like again, you can monitor all the, all the place. Um, and this was cool, but it didn't really like attach to the old world. So we still had the, the local agent loop running in vs code window. And then we had this new agent which was kind of doing its own thing. And so how do we bring those together? Um, so that's where we, where I would say like around February, March of this year. Um, and then like going from there, uh, something that also we wanted to be able to do was to, uh, be able to run agents, like even when your window was closed. And so if you have, you know, a long running agent that maybe, maybe have like a multi-step thing that they use, like hooks uses all these advanced features to be able to run like an hours or days long directory. Um, then you can't really have a window open on your machine on for that whole time, you know? Um, yeah. Um, and so then it became a thing. Okay. So we have, you know, we want to have this like long lived agent agent session something, something, something. Um, and so we, and so we need like, basically. And so we basically need a, a, a server for that. And we have like had the vs code server that if you've like used like mustache or like containers or w SLE, it's a component that we drop onto that remote box. And then and then we connect to it from our UI. And then that's how, how, how remotes work in 10s. Um, and so like, this kind of seemed like a natural sort of place to put that, but then got to the second problem in that, that physical server is actually locked to the version of code that you're using. Um, and this year we started actually doing also like the weekly, the weekly, the weekly like releases, which I know many people have probably seen that big blue update button in the editor every day, all the time. Yeah, yeah. Nonstop. Yeah, I love it. Yeah. Yeah. If you're on insider, it's every day. Um, but like, it's like, it's like, okay, you know, we can put it in the server and like, we can run the server, but that server is like commit locked to vs code. So we don't really have like a notion of like cross compatibility. It's just the server is built on commit that the client is built on. And then like when you connect to remote or when you start the server, we pick the server that matches that commit version. But if you have a session running for, say, a few days, which might not be common now, but you know, in like a year from now, you can definitely imagine models improving and getting better at that. Um, that will probably, or that will definitely be a thing. Um, and so we can't really say, oh, like you can't connect to this remote because the version is like too old or too new and like, you don't want to call that session. So then the next step from there is, is, okay, we probably need a protocol for this. Like we need like a separate like version protocol that we can use to, to interact, that we can point a client and say, okay, this client speaks protocol. It speaks to a server with this protocol. Um, and, uh, then as long as you know, we can do like, kind of like some verb, we can do versioning there, like independently of what we've traditionally done just with like IPC and vs code. Um, then, then we can do that communication. And that essentially is what the protocol is. Um, okay. So, uh, that's kind of how we got to like where we are, uh, currently. And, uh, like indigenous protocols is also built to, uh, more than just, you know, like talk from a like client vs code to a server. Um, because it's like, okay, we have the server, it's running on our mode. It's running, it's running a session that's running for maybe a few days. Uh, what if I connect from my desktop machine? But they also connect from my laptop machine. Like we have to have like coordination and kind of, um, and like there we have to deal with like disconnections. We have to say, you know, if I have a client that connects and maybe provides like a browser tool, how do we like model that? And how do we say, oh, I have my machine connected. I can, I can, I can visualize a browser. Even when I, even when my, even when my, my machine, we're running the session might be headless. Um, and so there's all these sorts of things that, um, that are ultimately like what we ended up tackling in the, uh, in the protocol. That's pretty cool. And it seems like if I kind of decouple it like you kind of are and you know the sessions out, right? You really are sort of taking it out of the process. Right? And I think that's a really nice way of evolving and looking at how agents work, right? Like you said, you had this window. You had a chat chat. It was it was there and it was gone. And then we kind of moved it out. It's up one layer across the the main window doesn't have to be open. The agent is aware of the different folders. It can do stuff. And then you get into this problem is, well, me, I'm on all these different machines or I'm online and I have tunnels and like, where and how are these running and how am I persisting? Like when you go to the agent host protocol website, it says synchronize multi client state for agent sessions. It's a very nice website. It's all open source. Actually. I think the website updated between yesterday and today I'm pretty sure um which looks very nice. So is the, the agent host protocol. That's something that we at Microsoft here created. I assume it's open source. There's a specification is the hope that right now VS code is using it and then others use it or what's the hope? Definitely. Yeah. So it's kind of like, like in the, like in the, the kind of lineage of the debug adapter protocol and the language server protocol, which are two protocols that, you know, the VS code team also like made, uh, that are now standardized, uh, as things across the ecosystem of Ides. And yes, this is yes, this is, this also kind of aims to be like a similar thing where, um, if you like, want to build, um, like a client to a like agent ecosystem, then you can use this protocol and then you can plug into any server that speaks protocol. Likewise, if you want to build like your own version of, of, of a super cool agent agent, harness like orchestrator, then you can then implement like that recall. And then you can, um, all of a sudden just like work with all the UIs that are built for it. So, um, like it's used in vs code, you can actually point vs code at, any server that speaks the protocol. Um, and likewise we have um a flag in our server. So if so if someone wants to pull down the server, um, or just um, if they're on their computer, they can run code space agent space host, they'll actually run a local agent host and then they can build a build a UI out of that. And actually, uh, we had like a few people on our team do that. We had some. We had like someone make a, a mobile app, like actually made a little process that just connects to my stream deck. And so I can like now monitor my agent sessions, like on my stream deck and I can like, see like if they're done or if, or if like it has like a question or input. And I can just click on the button on the stream deck to open it. And like in the agents app, go on, go on the code window. Um, and it's, it's actually quite pluggable. Like, I think that I think most clients that we've built with it, like the AI has been able to like the like model has been able to like One-Shot. It essentially just like point at the protocol, say, I want to build a client and then does it. And it's actually pretty cool to see that happen. So right now vs code and the agents window, how it's showing up today is it then both the server and the client basically in some instances, and we have multiple clients that are attaching to this agent host server basically if I'm saying that correct. Yeah. So by default we ship the agent host which is built into VS code. So it's going to be spawned as soon as you want to interact with it. As soon as you send the chat, either in the agent window or in your VSCode window, it's actually today, um, if you have a, if you have an agent who's backed, uh, like session just in your, in your workspace, you can actually open the window and see that happening like also over there. So, so that's actually, that's actually like already kind of like thinking locally, um, in that way, but also the, also the agent window actually supports connecting to like a stage in NWSL. Uh, It's probably going to come soon. And you can also just configure it with any like WebSocket like host port that, um, any server speaks on. Very cool. So that means like today, because I did a video on the VSCode dot dev slash agents website, which I think that that's using this agent host protocol, correct? Yeah, exactly. So that would mean that while the team builds VSCode dot dev slash agents, anybody could then build James's best agent sessions management tool thing. And that would be then speaking to all of the sessions that were then spawned from VS code. Or I could spawn them myself, I guess too. Uh, definitely. Yeah. Um, so that's just that website is going through the, the, the tunnel connection. And so, um, if you're in the vehicle window or if you're in the session window or sorry, the agent window, there's, there's a, there's a little radio tower icon. And if you click that, then that'll basically publish that instance to a tunnel. And so then you can connect to it from any other like window on, on any machine, like where you're signed in or just go on, on dev tunnels or VSCode dev slash agents and then pick it and then connect to it there. And then also like, as you said, um, if wanted to build like a super cool new, like iOS app to do agents, um, uh, same deal there. We have like a, like a swift, um, swift SDK and some folks have used that like already to build those apps. So it's pretty cool to see. That's awesome. Yeah, I think that's really neat to be able to say, hey, there's this thing that was locked in and it kind of really talks to the openness of VS code, right? This extendable ecosystem, the APIs, even for what's in the window and things like that are available for developers to tap into, get their models in the picker right from the drop down and now being like, hey, you can build these custom experiences on top of our ecosystem. So I think what's fascinating is when you go to the agent host protocol website, you kind of get this idea of, you know, talking about the problem of these sessions or stuck wherever it started. Now you can bust out of this, but there's this interesting image that I think kind of talks about how different ideas and different agents and different things are talking up to this agent host. And one of the other key pieces of terminology that I see in there is agent client protocol. And I noticed recently, like the copilot CLI, for example, can expose agent client protocol. And then like IntelliJ or like Xcode can kind of connect to it. What is the connection, if you will, between agent host protocol and agent client protocol? And like, is there any synergy there? Like, as far as like people that are building these experiences? Yeah. So the, yeah. So the agent client protocol is essentially like a control protocol for, for a harness. um, and be useful like you can, I think, um, uh, supports it. And then there's a bunch of like, you know, like, uh, like, um, ecosystem of like UIs that can control these things as well as like harnesses that support them. Um, but, but, but it doesn't really solve some of the more like high level things like how do we, uh, like sync state and how do we handle like reconnections and how do we do all these other kind of like, I guess, like hard things in the networking world and like in a more distributed system kind of thing, uh, like where we want to like retain the state, uh, and the clients to come and go seamlessly and handle like connections, handle, you know, timeouts, all that sort of thing. Um, also, I'm also the protocol, um, can speak actually to like multiple harnesses and actually in, in, in our agent host implementation, uh, we actually support, um, of course a copilot as well as as well as the cloud SDK. And also the SDK is going to come soon. And so you can actually use all of those like in that model. Um, and then like you can use like all those, uh, wherever you have an agent host, uh, like server deployed essentially. Um, and so it's kind of like a meta layer. And actually, I think something that that will probably do at some point is actually add a, built a built in connector for ATP. And so you can like configure your agent host to like point at this, like AC speaking, AC, AC speaking harness, and then it can just plug right in there and just come up as one more option, uh, give a copilot Claude and Codex. Oh, fascinating. So you almost kind of see this world where these harnesses for anyone can kind of invoke the harnesses, but anyone else can then spawn and manage and kind of kind of create an ecosystem of harnesses and sessions all living harmoniously together. Is that kind of the vision that we're seeing here? Uh, yeah. Basically. Um, yeah. So it's kind of like, so the point of view of it's the point of view of the actual protocol is basically to describe the UI that should be shown for a client that is consuming these things. Um, and so, um, we are intentionally like making it like very like, not like specific, like we have, you know, like vs code isms and like things that the, the, that like we add and things that like we need to do. But the protocol itself is like meant to be very generic and so that anyone can very easily implement it and like get off the ground and like, they don't have to necessarily see like our view of the world. Um, but as long as like they can speak like these kind of like primitives, then, then they can plug anything into it. That's really cool. Is there any, um, when, when, because you said kind of everyone's going to start to see this kind of host protocol rollout. In fact, it may already be there by the time I know it's there for me and insiders, which means it's going to be coming very soon. Like you said, by the time this comes out, what is the sort of impact for developers? Both, hopefully all good, but anything different? Like, is there anything that is the that we're giving up when we're interacting with these agents through the host protocol? Is there any like really unique scenarios besides what we've talked about that this is going to unlock for folks once they start using it? Yeah. I mean, I think like voting is a big one. Like, you know, previously the agent window hasn't been able to support connecting to like other remotes. And so that's like a big unlock as well as the kind of teleport like feature the VSCode does like such agents. Uh, I'm also like, we're actually seeing like some like adoption as well for like other parts of, of, of the GitHub of the GitHub like ecosystem. And so like you should, you should also like make like more integration there, like easier. Um, in the future. And like, I'm not like, I'm not sure how much we can talk about that yet, but there should be like really cool things coming down the pipeline there as well. But like overall, I think like if you're like, just like in, in, in like a window. It should, it should look and feel about the same, but as soon as, like you want to like go beyond a window, like when you start giving, like start to want to deal with like, you know, like work trees, uh, like remotes, like, like managing, you know, like dozens of sessions. Like that's really where, um, like this new model or this new view of the world, uh, pays off. Are we also going to see Connor if I open up vs code and I open up vs code insiders and I'm using the host protocol in each are my session is going to show up at the same time in both places, or are they still going to be decoupled because they're both running different servers? Uh good question. Um, I don't think that we have that out of the box just because like I'll probably is going to want to like ship ahead of stable. Uh, but you could definitely actually have, you know, like insiders just like, like point at saying, here's this host and port of my stable host server or vice versa. And then and then if you do that, then it should just work, you know, assuming the other versions are all compatible. Yeah, that'd be awesome. I mean, it does make sense because insiders is revving all the time and things are changing all the time. But it sounds like this decoupling would mean that I could almost build an extension if I wanted to, to kind of do whatever I wanted, I guess, and connect to these different host protocol servers and then pull them all together. Like there's nothing stopping me. If I was to run on like five different machines to create an extension that connected to all those machines and then show everything running on all my machines. Like that's a possibility, right? Yeah, definitely. Like you could like build like an extension for that. Um, you know, like, I think the, the, the, the point where the, the like support gets put in it all isn't a setting. And so you could, you know, like have, have your extension, like write that setting. And then when you do, it'll just all show up in your list. You could build your own custom CLI or like build like a WebView little thing in vs code. Like, um, there's, there's also, there's also possibilities. Well, let's talk about those possibilities. So you kind of talked a little bit about like the problem that it's solving the implementation, kind of how it's showing up today. And we'll be showing up inside of VS code, I guess, like, what do you think? Like long, long term, like these types of experiences. I've talked about a few things I think about how is the team looking about it. That VS code is going to evolve. Like once this thing is rolled out, like what things is it going to kind of enable? I mean, we're not going to commit to anything, but this is the VS Code Insiders podcast. So we can pipe dream a little bit. Yeah. Um, I mean, I think the next kind of like big thing is more around like orchestration. Um, if, um, for like those of you on insiders, uh, like you might have actually started playing with the, like the, the, the multi chats. And so like when we think of session, we actually like now have like the notion of like more than one chat per session. And then you can start thinking like that, you know, like obviously it's like, oh, you know, the, the, these chat sessions can start like sending messages to each other and start doing orchestration, things like that. And so I think that's actually like the next like big thrust for us. Um, and like, again, like, and likewise, like, and then likewise, that should also be modeled in protocol. Um, and so like, um, yeah, again, like, like, like, like prescribed, you know, here's how you do orchestration, like we should be able to represent that. Um, and then there's like a whole bunch of things there. There's like, there's like lots of companies and solutions that, uh, do something really cool and crazy things. But I think as an industry, we're, we're kind of like starting to align on those, uh, primitives that, that, that work really well for, for, uh, for orchestration. Um, and like adding those, those to vs code as well as to protocol and making them like really nice and reusable and, uh, and really flexible. Um, is I think what's coming next. Uh, personally, I think personally, and this is like not a promise or anything. Like personally, what I'd like to see, uh, as a developer is being able to actually like, kind of like strip my own like, uh, like interactions and like orchestrations. Um, um, I think like, I think like cron jobs are probably coming, but I want to be able to, you know, um, have like my own hooks that say, you know, like if this PR fails, I want to kick off at like agent and, and maybe like, do like five other things like along the way, like make me a cup of tea or something while I like, you know, like wait for builder. Stuff like that. Um, um, I think, I think this code has historically been like really, really good at, um, our story of like, of, of, of like, of accessibility and of like letting developers like, you know, build on top of our platform. Um, and I think that we have to figure out like what that means in, in, in the world of agents. Um, we, we have some of this, um, like we have, I think like APIs that people can like do certain things, like they can like add tools and skills, they can add agents and so on. Um, but I think that's something that like personally, um, I would really like to see happen is kind of like actually like letting more of the more of the user as well as the builders start to really dive into that and do their own, um, things there. That'd be really cool. Like you're saying, like kind of having a stream deck where you're kind of interacting with it. Or I can imagine like hue's lights or things that are, you know, you're, you're not just, it's, it's not just specific to one area or on your machine. It's running, but it's kind of like everywhere possibly available. I could imagine someone building some quite crazy Raspberry Pi like display showing all your agents happen in real time and like little indicators of things and where they're running. Uh, I think that'd be pretty neat. If people want to get involved, it's all open source, right? The agent host protocol, like if people want to get involved, like, what does that look like today as far as like the GitHub project? Yeah. So like, it's just like, and so it's still under like very like active work. Uh, like we are still, I think at, um, yet time speaking like zero point zero five or zero point five as, as a version number. Like we want to start to stabilize soon. Um, but currently, like we're, we're like adding more things. I think there are a few folks who already like dropped in and put in PRS and put in issues and like build clients. Um, actually like was on the, the, uh, like the MPM page for our, um, like TypeScript SDK. And I saw someone actually built like a, a cursor SDK integration for, for this. So if you want to use cursor from vs code like, like you can do that now. Um, um, and yeah, so definitely feel free to drop in, like start building on it and like, if you like, hit like limitations or if you have like ideas for something that you know, that doesn't make sense or could be better. I definitely just like if, if there's like issue or PR and um, yeah, that'd be awesome. Yeah. I love, you know, not only like, here's the problem it solves for us today instead of vs code, but here's sort of like the possibilities of what this could do to kind of really open up the rich ecosystem. Anything else you want to tell people about agent host protocol or anything else? Conor, before we, we pop out of this one. Yeah. No. Um, yeah. So, um, yeah, so it's coming soon. Um, I think probably by the time this out, it should be default. Um, yeah. Like if you have issues like, you know, uh, like on GitHub, we like, look at issues every day. So, uh, tell us if you have any problems. Um, and if you have any ideas, um, we can probably like add links to the, a project into the, uh, in the podcast description. Yeah. Most likely. And, um, and yeah, start building it and let us know what you create. So that sounds great. And the team's super open. Uh, I get to join the stand ups and I even send PRS and the team, you know, just accepted some PRS that I had or on different tool sets and all these other things. And it was really special to me because, you know, there's so much activity on the VS code repo. There's so much. But you know, if you write up a really good issue, put together a really good PR, you know, it's there. It's just like, it's cool to see things come and have such an active community around such an awesome editor. And even now, beyond the editor itself, which is so cool. Well, Connor, thank you so much for coming on and talking about agent protocol. It really helped. You know, it's showing up. So I'm like, what is the thing I really need to get the expert on? So I really appreciate you coming on and talking about not just what it is, the problems it solve and where it's going in the future. So thank you so much. Yeah. Thank you. Awesome. And thanks, everyone for tuning in. Of course, you can find us everywhere on the internet, VSCode podcast dot com. You can find all of our episodes with the amazing team that builds your favorite code editor. And of course, check us out on YouTube, YouTube dot com forward slash code. Of course, the video is there. I'll put little things there and you can find all the links like Connor said in the show notes below. So until next time, I've been James Montemagno. This has been the VS Code Insiders podcast and as always, do you want to say it? Connor. Uh. Happy coding.