Daniel (00:01.006)
Hello. This episode of Waiting for Review is sponsored by CrowdStrike. Decelerate your life by getting away from your computer every now and then with CrowdStrike and Microsoft Windows. Touch grass and just ignore the blue screens for a while. Register now and get a $10 Uber Eats coupon for free. CrowdStrike, decelerate your life.

S4e25 (00:23.628)
Don't open doors, break some windows.

Daniel (00:28.782)
Hey Dave, it's fantastic to see

S4e25 (00:32.29)
Good to see you too, Daniel. Were you crowd stricken? Did anything hurt you in any

Daniel (00:38.262)
I don't think so, no. I don't run any Windows machines either privately or with the job. well, I used to host a lot of machines on Microsoft Azure, but those were our Linux machines and they were fine. I did have a short database audits last week, but I don't, that was kind of coincidental from the timing, but I don't really think, I can't.

Like I can't say anything. The database was hosted, thought it was on Azure. It was gone for 15 minutes and then was back up. I don't know. I don't think I was crowd stricken. just like, that's why I can like chill. It kind of can be so fun about it. Like I did, I wasn't on any flights. I didn't take any trains that day. Like I only like saw it on the news and in the internet. That's perfect.

S4e25 (01:13.773)
fair. Fair.

S4e25 (01:25.227)
Mm -hmm.

S4e25 (01:31.816)
Same, same to VFIR. Quite cozy in my little macOS, Linux -y bubble. no, and also simultaneously like it, I think it would only affect you if you had paid for CrowdStrike services for your thing. Yeah. Yeah. But no, no, interesting. It's certainly interesting to see that in the news

Daniel (01:48.12)
Yeah, that's the other thing, which I have not.

S4e25 (01:59.02)
Obviously my heart goes out to anybody actually impacted by it. Because these things are not cool when you're stuck somewhere. And you're stuck somewhere because that place is using Windows. And in a way that I would describe as holding it wrong. But anyway.

Daniel (02:20.184)
Yeah, every now and then we complain about stuff that Apple is doing, but at the same time, for me personally, Mac OS is still the best of the operating systems. just fits very well with how I use computers. And so I'm very happy to be using a Mac.

S4e25 (02:32.854)
Mm -hmm.

S4e25 (02:36.298)
Yeah, indeed, indeed. No, no root kits for security over here. But moving swiftly on, you mentioned servers and things, Daniel, and I definitely want to ask you how your server move has gone. But to put me at ease, because you know how antsy and nervous I get, do you want to head into the intro?

Daniel (02:59.47)
It's been a whole minute or so. You must be like really tense. So let's get straight into it. Hey, welcome to Waiting for Review, a show about the majestic indie developer lifestyle. Join your scintillating hosts, Dave and Daniel, and let's hear about a tiny slice of their thrilling lives. Join us while waiting for review. I should have taken a screenshot of your jazz hands right now.

S4e25 (03:27.426)
For listeners of the show I was making jazz hands and other movements as Daniel read the intro, largely because I wanted to see if I could make him crack.

Daniel (03:40.029)
I did smile more than usual because usually I'm a very serious

S4e25 (03:45.174)
Serious boss scintillating,

Daniel (03:47.724)
Yes, yes, as we Germans are wanted to

S4e25 (03:52.418)
Indeed.

Cool, well, I did. Yeah, how is it? Have you moved?

Daniel (03:56.29)
You had questions about the server moves.

Daniel (04:02.766)
So last time we spoke, of the used by Telemetry Deck stood in Amsterdam. And as of right now, almost all compute capacity that is used by Telemetry Deck is being used in Frankfurt, in Germany.

S4e25 (04:25.64)
Ooh, yeah, that's good.

Daniel (04:28.258)
Yeah. I have moved almost all my servers. And as of now, they are converging more and more on stable, which they've been for a while. then it's just about finding the perfect configuration and then throwing different things at them and see if they buckle. And so now I've kind of run out of things to throw at them.

And so I haven't found anything else to make them really crack. So I'm about to remove the notice from the website that says, something is like we're moving servers, so excuse any disruptions. Yeah, we moved servers because we had them on Microsoft Azure, which is a lovely hosting provider. But we moved to Amazon AWS because they basically gave us a lot of free credits.

whereas the Microsoft credits kind of ran out and at the same time, it was very advantageous because I wanted to re -engineer the architecture of the whole server landscape that I had as well. So I would need to, I needed to like rebuild everything anyway. And so this was a good, good time to do it basically because, so telemetry deck has various parts, right? And the,

The two parts that are very important is where the data comes in, into telemetry deck, and then also where you get your queries out, where you send a database query or a telemetry deck query language query, and then you get some kind of smart and intelligent and insightful answer back. And so the data goes in through the ingest API.

S4e25 (06:11.67)
Mm -hmm.

Daniel (06:26.626)
That's at nom .telemetrydeck .com. And then it goes into a message queue and from the message queue, it goes into the database server driven by Apache Druid, which I've talked about a lot before. And then like there's a separate API, the query API that also powers the front end. So it has like a normal database attached to it as well that does like user logins and I don't know, stores all the credit card numbers and

S4e25 (06:27.126)
Mm -hmm.

S4e25 (06:31.125)
Nom nom nom nom.

Daniel (06:56.11)
the social security, I'm joking of course, we don't have any credit card numbers. Yeah, and so the main cost driver is of course, the pretty humongous Druid cluster that used to be on Azure and that used to be about, I wanna say 15 to 20 machines. And because now it is way more efficient, I've...

taken it down to eight ish depending on how you count, even a bit lower. And the reason for that is I moved away from Kubernetes for that specific workload. I have actually come to, I don't know, I think I like Kubernetes now. I get what it's good for and I like the ease of use that it brings for certain types of APIs, for certain types of workloads.

which basically is, yeah, I build an image, a Docker image. Is it Docker image? My brain is kind of fried. I've had a long week. I build a Docker container. That's the word I was looking for. I build a Docker image, and I just throw it on there, and it just runs. And I can

S4e25 (08:11.714)
Keep going. Yep.

Daniel (08:18.306)
distributed and whatever and then if I like release a new version of the thing it just builds a new image and then I tell the thing here use the newest image but then I can also roll back to the previous one and that's pretty cool. So the problem is with druid that it needs a humongous amount of RAM and CPU and it really kind of wants to be the only thing running on a machine because

because it's written in Java and so it kind of wants to reserve in advance. It wants to know how much RAM is allowed to use and then it will kind of reserve all that RAM in advance. Not always, but most of the time. And so that means that I can't really do that on Kubernetes because even underneath Kubernetes there are actual machines and I can

change the size of that machine. in the previous Kubernetes cluster, did have machines that had, I want to say, 32 or 64 CPU cores and then 128 gigs of RAM or something like that. But then the thing is, what happens is I can't really max out, especially the RAM usage on these machines. Because as soon as of those 128 gigs, as soon as I am my AA or

the sum of all Docker containers running on that machine exceeds, I wanna say, 100 gigs or so, the machine will actively try to offload stuff because it's like, I'm feeling a bit squeezed, right? And so I've had various techniques to make sure that only one image or one container is always running on one machine. And so that container can

use the whole 100 gigs of the 128 that I'm paying for. But still, sometimes things were just like, the machine would just decide, yeah, I'm just gonna kill that container and whatever. Then the other thing is like, it's kind of hard, it's not impossible, but it's kind of hard to have a good, very quick hard drive attached to that machine because it's all virtual, but then you have various layers of virtualization.

S4e25 (10:15.042)
Mm -hmm.

Daniel (10:38.158)
And so most of the storage is kind of networks attached, which is kind of bad for these kinds of workloads. then, so out of the 128 gigs, I only have a hundred left. Then inside the container, I also have to have another bit of buffer. inside the container, I'm like, okay, this Druid node inside this container can only use, let's say, 80 gigs of RAM, because otherwise it will just

touch the upper limit and then just like completely destroy itself because that's just how it works. And then that is kind of the main problem. And then also the implementation of these things is kind of a bit not very performant, I want to say, or not very efficient. So you end up provisioning or holding ready a huge amount of RAM, like at least twice as much as you actually need and paying for those machines, of course.

without ever being able to really use it. And at the same time, if some process is being killed or whatever, because these containers then get spun up in their completely fresh environment, everything is ephemeral, they need to reinitialize, they need to get all their data from storage to be able to go back. it's not like a machine that's rebooting and then it's in the same status

And like we've even had like listeners like writing in and also advising me on like, get off Kubernetes with that kind of workload. That is not what is happening. And that I've been planning. I've been planning that for a while and just have been trying to get all the puzzle pieces in place with the AWS transition, but also with just like finding people who can help me and or give me advice, like people like.

S4e25 (12:14.068)
Yep.

Daniel (12:33.102)
I can call sometimes and be like, hey, can you look at this because I don't get it? it's like, what is a swap off command? Why does my virtual machine that should have swap space not have swap space? You can enable it. Swap off, mate. There's your episode title.

S4e25 (12:52.172)
Sounds like an insult that's like swap off,

Yeah.

S4e25 (13:01.332)
I guess so. You what? Swap off.

Excuse

Daniel (13:07.342)
So yeah, one thing that I really wanted to do is I wanted to, like, because what you can, of course, do is like, can open AWS in a web browser and then click on new machine and then, I don't know, like enter all the details you want to have with the machine and then click on new machine again until you have like all the 10 machines that you need. And then like SSH into every single one of them and then install all the software. But that feels very brittle. And so.

My buddy who helped me with a lot of these things, he says treat servers as cattle, not pets. So what we did is we used a software called Ansible, which is a toolchain to provision servers basically. So what you do with Ansible is you write basically a series of commands and their start state and their end state.

And then it will just basically run them on all the servers with some variables and some templating and some, you know, if and and buts and whatever. But the fun thing is because you, these commands usually have a defined end state. You usually just tell them, tell them the end state and then it will do whatever is necessary to reach that end state. Like for example, the state would be a service called druid with this kind of configuration file.

is installed and activated and running. And so the fun thing about this is always, it's also like, I can run this command multiple times. And if the service is already there and if the configuration file already contains all the variables that I've given it, nothing will change and it will not do anything. And so what you end up is you have an Ansible playbook. That's what they call it because it's very like a theater metaphor. You have like roles and plays and

And so, like, I have roles for, I don't know, machines with Java installed, and then machines with ZooKeeper installed, machines with Druid installed, both kind of depend on Java. And then I have roles for individual types of Druid nodes because those need a bit more configuration and stuff like that. Also, like, a lot of monitoring. And what happens is I can change the configuration.

Daniel (15:33.452)
And then it will just run on the CI and just run all these commands. it will, like you connect to a, what we call a jump host, which is like accessible from the CI. then from that, once it's in there, will SSH to all the different servers. It will find those by asking AWS API, like, Hey, give me all the servers with a label that is called, I don't know, Druid middle manager. And, and then we'll run all the, all the tasks that I drew at middle manager.

needs to have. So if something changes, those will change, but then all the rest will not change. And so another work was, yes, question?

S4e25 (16:05.012)
of

S4e25 (16:13.234)
Yeah, yeah, yeah. Can you use this in such a way that you've got very clear like staging servers or rather a test environment as well? I think,

Daniel (16:25.176)
Yes.

You can test this in various ways. Like, of course, actually they have a linter and everything, which is very cool. And I kind of enabled that very, very quickly. It's all Python based, by the way. And what you can also do is, like, you have what is called an inventory. And so that is kind of the list of the servers. And so my inventory is a dynamic inventory that is created by querying the API for, like, hey, give me all the list of the servers with this label.

And actually before that step, it will use the API to create all the servers that it needs because I already also define all the VMs in there. And so if I have just like an empty field, will like do everything it needs to do that it has like 10 servers, some of them smalls, some of them big machines, and they are configured in that way. And then it will kind of refresh them, wait for them all to be available and then run all the tasks on them.

And so what you can also do is like have a separate inventory for like a test environment or a dev environment. What you can also do is like you can run a diff. So you basically run the playbook, but you're not changing anything. You're like, you're just asking to go through all the steps and then at the end report how your inventory, your list of service would look differently.

S4e25 (17:36.234)
it.

Daniel (17:57.038)
if that thing ran. And so that you can also run that, that like in, I don't know, in pull requests, for example, it's just like see how, because right now, in fact, I don't have a staging environment. I have all the pieces in place to run a staging environment, but right now, it is a lot about, or like the last week, I want to say, there was a lot about finding the perfect configuration so that these machines would

S4e25 (18:05.036)
That's useful.

Daniel (18:26.794)
run very efficiently, as in they would use as much RAM and CPU of the machines they are running on as possible without being unstable. And those are things that I can only really check with the exact machine sizes that I have or that I want to use for production because those are the ones that have CPU cores and the RAMs and everything in that configuration. And also with the full telemetry data set because that's kind of

What's tricky because I didn't even realize, but the uncompressed data set is nine terabytes now, because I had to copy. That's the amount of data I had to copy from place A to place B. And so only if you throw all that data on all these machines and then give them various different types of tasks, sometimes at once, like, hey, clean up your data and also give me an export of this.

S4e25 (19:06.069)
Yeah, yeah,

S4e25 (19:10.018)
Wow

Daniel (19:25.132)
Most everything runs right now. that are things like I could only try those out with the live data. So at some point, we kind of built it up. And then we kind of ran it. It didn't work. And then we tried out different things, did some debugging. But because it's infrastructure as code, you can use PRs. You can use and.

S4e25 (19:44.169)
Yeah, yeah, yeah.

Daniel (19:52.556)
It's not like you're clicking around and just trying things out. You can actually have a ticket and then attach a PR to that and then try things out. And if it doesn't work, you revert it and stuff like that. And so that's kind of helpful. So yeah, we ran a bit in parallel. And then after a week, so last, I want to say last Friday or last Saturday, it was stable enough that

S4e25 (20:03.596)
That's cool.

Daniel (20:22.947)
I basically switched over to the new servers for a few hours. Turns out they also were fine with all the requests coming in that were about querying stuff. I've had the signals coming in. That message queue has been split for long while, sending to both of the stacks, to the old one and to the new one. Because that's kind easy, and I can just do that.

Where do the queries go to? That has to go to one or the other cluster. And yeah, a few days ago, I switched over. And it's been mostly fine. I think I found a bug in exports that prevents them from running sometimes. if you want to export your data from Tendemintutica right now, that might not run perfectly. But other than that, I think I have all the data.

have all the data copied over and included in the new cluster. It is way more performant, which is really nice. And it also can do a thing called compaction, which is kind of like defragmentation. I think I talked about this before as well. And the previous cluster was so RAM -constrained that it couldn't really do this because it would kind of fall over whenever I told it to defrag a day. And now this one with less machines has been just

plugging away this and it's all like it's already like has already like the defract or compacted about 2 % of the data set. And it goes like newest newest to oldest. So the benefits are coming immediately. And so in like in a week or so it will probably have like have compacted the first or the newest month or so. And so then the performance should even increase.

S4e25 (21:57.794)
That's cool.

S4e25 (22:16.002)
That's awesome. That's really, really awesome. And I love what you're describing in terms of config as code and all of that side of stuff, because it means you can be really surgical about your changes, right? It's not just like, I've got to write down every step I've made or log everything together or whatever you're going to. Yeah. Yeah.

Daniel (22:33.495)
It's all in good.

That being said, one thing that is really exhausting about server work is just the fact that everything takes so long. Like, I want to try something out. It's not like pressing Command R and then it compiles in a few seconds. It's like, I changed something in these Ansible playbooks. And it takes, I don't know, five to eight minutes to run.

just until all the servers are reconfigured and restarted and everything. And so you're sitting there for eight minutes and are like, I can't really do anything else because I will lose the stack of the house of cars that I have built up in my head, like the next three steps that I really want to try out. And I'm just trying to confirm one suspicion that I have. For that, I need to wait for this before I can continue to go back to the file and try stuff out.

And so it's been a lot of hurry up and wait, and that is weirdly exhausting in a weird way. Like, because usually when I, when I really hyper -focus on a technical problem, like my brain is like going full tilt, like I'm working so hard or like I'm solving so many little problems one by one. And here's like, it's like the brain, my brain is completely active and I need to be very present, but at the same time it has these.

S4e25 (23:35.82)
Mm -hmm.

Daniel (23:59.982)
weird waiting periods. so it's very hard to stay motivated and concentrated. But at the same time, it's very hard to get away from it. Because I feel like as soon as I turn away from the computer, something will happen. And so I'm all like, you just sit there. And I had these really insanely long days too, because I'm like, this really needs to happen.

S4e25 (24:26.698)
I know, yep.

Daniel (24:29.549)
this needs to happen this week. have deadlines that I really wanted to hit. so, yeah, you're just sitting there and suddenly it's 11 and it's like, I haven't really done anything. it's like, most of the time I've been just waiting for stuff to finish provisioning or,

S4e25 (24:46.178)
Well, I mean, you described the round trip and development there as being sort of up to like eight minutes or so. I mean, like even if you were to, if you're okay. So if we were to say that it was five minutes around trip, yeah. Then you've got 12 of those in an hour. And that's not a lot if you're trying things and if you're like, but what about this? But what about that? Yeah.

Daniel (25:12.81)
yeah, and also like that is just the servers. Like then I also need to do stuff on those servers, right? And with the amount of data that we have, like it's more like, okay, I'm waiting five or more like eight minutes for everything to reprovision. And then I'm like, okay, I'm now I'm going to the server and I'm telling it to rerun the task that I'm trying to run on it, which is for example, June 24. Just like load it and compact it or whatever. And it's like,

I'm going to do that and it's going to distribute all these tasks on the whole cluster and you see all the CPU spikes and then it just stays that way for a day. Ingesting a day takes about 40 minutes. Sometimes last week I actually increased the amount of servers' temporality to a lot more because of

Yeah, I'm not waiting that long. So I could actually in just a day, like 15 minutes or so. But even then, if you're loading a month, that just takes time. But it might also crash at any given moment, because I'm still debugging. What is the correct? This software has so many knobs and dials and everything. And I can configure everything. And it's like, there is a correct configuration.

S4e25 (26:32.044)
Mm -hmm.

Daniel (26:38.614)
It won't tell you what the correct configuration is, but if you're not setting the correct configuration, it will be like, jail.

S4e25 (26:47.402)
Yep. man. I'm never going to complain to you about Xcode's network debugging round trip now because it's certainly an easier ride than you've just described.

Daniel (26:49.95)
Daniel (27:06.646)
At the end of the day, I have collected so much experience with running a Apache Druid cluster by now over the last few years that you get a sort of intuition, like where to look and stuff like that, and also how to debug stuff. But still, I think that, okay, this feeling that, okay, this task is gonna take two hours, but it also might crash at every second, and I wanna react to that

I can't like, this is such a weird thing because you feel like chained to the computer. Like, you're like, yeah, this, can't really do anything else because like it actually literally happened like a few times I was like, okay, I'm just gonna, I don't know, like clean the apartment, go for a ride on the bike or whatever. And then like, I come back and like three minutes after I, after I close the laptop, it's like, no.

S4e25 (28:04.572)
man.

Daniel (28:05.484)
But at the end of the time, I learned a lot and also I'm really happy that this was not perfectly smooth, but actually way smoother than I had dreaded

S4e25 (28:20.94)
It's cool and I imagine like this work as well it means that if you do want to do this sort of a move again in some fashion, you're a lot better set.

Daniel (28:30.114)
Yeah, that's exactly why I chose Ansible and not for example, some Amazon specific proprietary format, because I can imagine, for example, that in the future when like machine prices are more impactful and we have like, let's say we, we, we hire a DevOps person and I can just like give this whole load of work. can just give to give to them and then they can say,

S4e25 (28:54.134)
Mm

Daniel (28:59.438)
Hey, I have more time than knowledge than Daniel. So I can just spend a bit more time and make this 1 ,000 times more efficient on bare metal servers, let's say. This is totally possible because this is very agnostic.

S4e25 (29:10.178)
Yeah. Would, would, would it let you support like a multi -cloud kind of scenario as

Daniel (29:21.28)
I mean, yes, but what needs to happen is the different nodes of this DREWD database. And by nodes, I mean machines. They need to be able to find each other. And to do that, they use either host names or IP addresses. So the problem is it's definitely possible, but you need to find a solution where you securely poke holes into the different

cloud providers firewall so that they can find each other, but not be accessible from the whole entire internet because that will be kind of slightly unsafe. There are, I don't know, like this thing called tail scale, for example, that a lot of people are talking about and we kind of experiment with that a little bit, which I don't really understand yet, but it feels like VPN meets BitTorrent, whereas

S4e25 (29:57.1)
Yes.

Daniel (30:16.758)
not really a server, but somehow they kind of find each other and it's like all encrypted and you need kind of the correct certificates. So something like that would probably work. But right now it feels like more trouble than it's worth. But yeah, this would totally work if you just find the correct configuration or like connect the networks in the right way.

S4e25 (30:33.186)
That's fair, that's fair.

Daniel (30:46.056)
I could even like put put one machine here on my desk or something like

S4e25 (30:49.794)
That'll be fun.

Daniel (30:52.824)
Yeah. Yeah. So I talked about the Apache Druid, which is basically our main database. And also about the different, I also mentioned the message queue and I mentioned the two APIs, the ingest API and the query API. And actually the message queue also moved over to AWS. But right now the ingest API and the query API, they still live in Azure. They still live in the

Kubernetes cluster. Because actually, I did create a tunnel, but an old -fashioned VPN tunnel, so that those can actually query the new servers and also send data to the new servers. But by leaving them alone for now, I kind of bought myself some time.

S4e25 (31:24.33)
Right,

Daniel (31:46.336)
I could build up the existing or the new stuff pretty well without like if everything just falls over, could like switch back to the old solution. So at some point in the next week, I'm going to have to spin up another Kubernetes cluster.

And then move those as well. And then at some point, I'm going to have to switch. I'll have to update DNS entries. And I'm so looking forward to it.

S4e25 (32:12.843)
DNS is always fun.

Daniel (32:15.618)
Yeah, because I have so many fail saves in that whole pipeline where signals arrive at the API and then they immediately get stored to disk and then they get processed and then they get stored to a storage bucket on Azure. Then they get put into

the message queue, and at the other end of the message queue, they arrive at the database, but also they get written to another bucket. So if any part of this fails, things are still there, and I can reload them. But DNS changes. There's no safety net here. So I need to be very careful.

S4e25 (32:49.164)
Just in case.

S4e25 (33:03.286)
No, not at all.

Yep, yeah for sure. I don't envy you on some of these large server scale problems. I sort of feel

Daniel (33:13.964)
Yeah, I feel like the time sensitive changes because of when credits are activated and deactivated and stuff like that. So that the move is not costing an incredible amount of money. Like every now and then I look at these prices and I think like, okay, that's like about two iPhones a day. That's pretty good.

S4e25 (33:37.85)
Yeah.

Daniel (33:40.822)
so to not spend too many iPhones per day, like the, this, this move was kind of time sensitive, but the rest is way less time sensitive because I don't know, like that's the, the, were more, more talking about like, I don't know, milli iPhones or a hundredth of an iPhone. and so. Yeah, I'm gonna like, I'm going to try to be very methodical about this and very like very slow and

like test and verify and stuff

S4e25 (34:12.62)
Yep. Yep. Now that makes sense. I'm trying to, I'm just that whole perspective though of milli iPhones and, and, yeah, it's amazes me just how big everything has gotten, over the last few years for telemetry deck. yeah, they really are. I, I was going to say, I saw your post with the new office that you've got for telemetry deck as well.

Daniel (34:28.406)
It is, yeah. It is. Things are moving. Yeah.

Daniel (34:40.29)
Yeah, like you are looking at me through a different camera too. Because I moved the big display that my normal camera sits on top of. I moved that into my new office. And so you're actually looking at me through a laptop camera.

S4e25 (34:48.246)
Mm

S4e25 (35:01.451)
wow. Okay.

Daniel (35:02.466)
Yeah, I lowered my chair so that the angle kind of works similarly.

S4e25 (35:06.614)
So yeah, the angles good. I was going to say you got me there. You're not looking too different in that respect.

Daniel (35:11.95)
Yeah, so here in Augsburg, there is a fantastic startup accelerator program. And if I say like if I say startup accelerator program, most people think very commercial, like, there's a huge company that says, are we gonna invest in all the startups and kind of just like get arm get like squeeze them for as much as possible. But this is actually paid for by the government, like the city government.

is saying like, hey, we need like people who are creating companies. So let's help them. And so this is an organization that is whose job it just is like to help people who want to create a cool innovative company, just help them with advice and also sometimes other types of help. And so they have recently expanded. They bought a huge old like container warehouse basically.

and they renovated it and they put in all these little tiny office cubes, like on two stories. It's pretty nice. Someone on Macedon said, like, looks like if Johnny Ive designed a paintball arena, which I can't stop thinking about. I can post, I posted some links, posted some images on Macedon and it's pretty nice.

S4e25 (36:14.89)
Mm -hmm. Yep.

S4e25 (36:33.462)
Yeah, yeah, that.

Daniel (36:37.44)
And so, yeah, this is really cool because A, it's actually very COVID safe because they have this really sophisticated air filtration system where they basically pump fresh filtered air into the individual cubes. And then the cubes have these little slits where air just exits into the main area, but the main area is also felt filtrated air.

S4e25 (37:03.688)
that's awesome.

Daniel (37:06.008)
But especially if you go into your little office, you just get the nice fresh air. And also it feels very open and friendly. it's not too much right now because they're actually not officially open yet. But because we're friends or friendly with people who run this whole place, they kind of ask us, how about, do you want one of those before the big run begins, before everyone gets one? Because they are.

S4e25 (37:06.54)
Yeah,

S4e25 (37:12.268)
That's cool.

S4e25 (37:20.748)
Mm -hmm.

Daniel (37:35.554)
way cheap because you only pay like a tenth of what like this kind of office space would be on the open market. And there's a few like, I don't know, there's a few caveats. Like you can't be on the market for more than five years and you kind of have to be from the area and you have to be technologically innovative and stuff like that. we completely comply with all of those, right? And so me and Lisa were like, huh.

I, we were like, we were like pretty tempted because it's especially for me, it feels like I have, I love working remotely, right? I've been doing it way before the pandemic hit even like, but especially recently, I felt like I'm kind of like a bit stuck at home and I don't, I don't get out at all, which is not 100 % true. I do get out sometimes, but like, I don't know.

I felt like it would be really nice to actually switch it up a little bit. so we were like, okay, this is like, we can just like try it out for a few months, see how it works. And so on Monday, we kind of went there and just had a look. Like they just like gave us a tour and they like, okay, it's this much and these are the features. like, yeah, it has height adjustable desks and like has this really nice office chairs and it has...

cable -based internet, like it has Wi -Fi too, but like you get like your own IP address and everything. And then Lisa, I was like, how do I convince Lisa to just like do this? Because I really want to do this. Like, I really want to try it out. And I turned to Lisa and Lisa just like looks at the guy and is like, okay, can we just sign now?

S4e25 (39:07.266)
That's fantastic.

S4e25 (39:27.842)
Ha ha ha!

Daniel (39:28.494)
And he was like, he was like super perplexed. He was expecting us to go like to basically leave and then call him a week later or whatever. And we were like, yeah, like we have our laptops here. Like we have some work to do. we just like, can we just come in and say we want that cube, please. Can we just go there? And it didn't work. It worked perfectly like that. What happened was they were like, okay, we need a few hours to prepare the space and to activate the

S4e25 (39:35.946)
Yeah, yeah, yeah.

S4e25 (39:45.026)
That's brilliant.

Daniel (39:59.074)
the network slots and everything, and just do bit of cleaning. And also they have these transponders to open the doors. They don't have keys, they have these little fobs. And to program those and whatever. And we were like, OK, let's just get some lunch. And so we went to a food truck that sold döner and talked about fun things

S4e25 (40:11.191)
Yeah, yeah.

Daniel (40:28.326)
US elections and fundraising and stuff like that. And then we were like, okay, let's like, enough time has passed, let's go back. And then it started raining so hard. Like it was an absolute downpour. We kind of ran, we were like drenched to the bone. So like first, first few hours in the new office was basically like we got merch t -shirts from

S4e25 (40:30.306)
Yeah.

Daniel (40:56.482)
from the organization. And of course the air filtration stuff also meant that the air is very dry, so like stuff dries. Also basically the first day we just sat there in our underpants and these t -shirts, these super large t -shirts, know. I just be like, this is very professional.

S4e25 (41:20.482)
man, I was wondering where this story was going and I did not expect that as the outcome Daniel, but

Daniel (41:25.486)
Yeah, we didn't last the whole day. were like, okay, let's like, we did some work, but like, let's go home and like put some dry clothes on. But what did happen was like, I went home and then I had a shower and like, got some fresh clothes and I was like, I wanna, I wanna like, I wanna move my stuff there. I wanna move, especially the big display. I wanna move the XDR there. I wanna like really commit.

S4e25 (41:32.641)
Yeah,

Daniel (41:54.04)
to this office and like really go there. And so I took the car. I filled it with all the stuff that I wanted to have there. I actually, I also called Lisa and was like, I'm drive the stuff there now. Do you have anything, anything like do you want me to come to drive by your place and like pick stuff up? And she's like, wait, but didn't you want to do it on the weekend? Like what happened? And I'm I'm just really enthusiastic about this.

S4e25 (42:09.634)
I'm in. I'm all in.

Daniel (42:22.796)
And she's like, novelty.

S4e25 (42:25.378)
Mmm, yep.

Daniel (42:26.498)
She knows what motivates me. So yeah, I fill the car with a huge screen, my keyboard, my trackpad, various network equipment, a few basically just knickknacks to have a bit of personal touch. I got gifted a few years ago, a long time ago actually, a Sphero, which is

S4e25 (42:47.126)
Yeah, yeah, yeah.

Daniel (42:55.766)
you this Bluetooth connected robot. It's like a sphere that you can like remote control with your phone basically. But this one is a special edition that looks like BB -8 from Star Wars.

S4e25 (42:56.01)
I know, I remember Sphero.

S4e25 (43:07.641)
That was the best one.

Daniel (43:09.39)
And so I was like, I'm just gonna like this. basically I have all these all these decorative decorative things

S4e25 (43:14.54)
Did you take that there just so you could run it around the new place? Have you been running this sphere around?

Daniel (43:19.426)
That I have not yet because it's actually like, like the batteries drain so much, like it doesn't turn on yet, but I put it on the charging pant overnight. So it's going to be, it's going to be fine tomorrow, I hope. but yeah, that was my thought. Like basically I looked around and was like, okay, the knickknacks that I have like standing around here, like I like them here, but I have a few things that are just like in the, the, in the cupboard that are like, yeah, they don't have space right now. And I look at, look in there and I'm yes, that's that I'm going to take

yeah. And so, yeah, we've put up everything. And so today is actually the, was actually the second full day where I've been just like working at the office. The coffee is pretty decent. I can bike there, which is, really good because I feel way more energetic and yeah, it's very nice.

S4e25 (44:10.028)
That sounds lovely. That sounds really cool. I vicariously, I want to see more of this space. So I think I'm going to ask you, could you pull together a very quick like TikTok or something to just visually show me around? Like, and can I?

Daniel (44:31.192)
Sure, sure, I'm gonna go there again tomorrow so I can film a quick walking tour.

S4e25 (44:36.043)
Yeah,

S4e25 (44:39.498)
And this episode won't go out before that's happened. So hopefully we can link all of that up in the show notes as well. People can take a look. Yeah. Yeah. Awesome. that's great stuff, Daniel. I'm really pleased to sort of hear how everything's kind of coming together at the moment for you, cause you've had long days. There's been a hell of a lot going on. yeah.

Daniel (44:45.541)
yeah, totally. That's a good idea. I like

Daniel (44:59.169)
Yeah. Like last week, I do, I track my time, right? Like last week I was like 65 hours.

S4e25 (45:07.21)
Yeah. Yeah. it's, it's a lot. I think, yeah, it sounds like you're coming out the other side of all of that now and into a new space. I can totally understand wanting to sort of hit the button on that as soon as possible. If you've been doing these long days and you've been, it's kind of, I don't want to say stuck at home. That kind of is, is to me, feels like the wrong phrase. It's more a case of like your home space and your workspace is getting

all blurred together, right?

Daniel (45:38.85)
Yes, exactly. like, this is actually a breath of fresh air. Like I come home and I'm thinking way less about work, but at the same time I'm sitting down at work and I'm like, yeah, this is, I kind of can put aside the fact that I don't know, I need to clean the bathroom and the hallway and

S4e25 (45:51.191)
Mm -hmm.

S4e25 (45:59.584)
Yeah. Yeah. So you can compartmentalize bits of your life more effectively with this. So I get that. I get that. And I think, you know, that for me being somebody who's working pretty much a hundred percent remotely these days, the, the, one of the ways I've been able to do that without going too crazy about it is that this house that we moved in last September,

I've got my own home office so I can shut the door on it and I can be like, okay, yeah, that's, that's that done. yeah, but, I think without, yeah.

Daniel (46:32.533)
Yeah. I mean, I have that too. Like I'm in my office right now, but I don't know. It's just like, I don't know. I mean, that's why I made it for so long. And I'm not saying I'm gonna be in the new office 100 % of the time. I'm not gonna promise that. But right now it's just like, it's just nice. It's just a nice different style. It's very reachable for me. And also no one is forcing me to go there, right?

S4e25 (46:57.536)
Yeah. Yeah, it's a big difference. Yeah. Yeah, for sure.

Daniel (46:59.136)
I can just decide to go there, which is also a huge difference, I feel like. Lisa, for example, says she's probably going to be there two to three days a week, whereas I'm targeting more like four to five. So yeah, sometimes she's going to be there, which is also a new experience for me, like working multiple days a week together with her. That's also very nice.

S4e25 (47:19.372)
Yep. That's really cool. Really, very cool. I'm quite pleased for you. Like this continual growth and phases and everything of Telemetry Deck. I mean, I've been with you on this since it was an idea of yours and each stage, I'm just like, hell yeah, go for it.

Daniel (47:45.035)
Thank you so much. We've been talking about me for an hour, I want to say. Do you want to give a quick update of the things you've been working

S4e25 (47:51.67)
Nearly, nearly, because we obviously we talked before.

S4e25 (47:59.434)
I don't want to get deeply into it because I think we've probably got a show here and I reckon we'll talk more about my stuff next week and that will give me bit of pressure to get some stuff done because just to touch on it very quickly, the app that I've been working on for like the last couple of months, this networked video switcher, I think I really am on the releasable version of it. So, I mean, I was plumbing

the last couple of bits, one of the very last things I tend to do is wire in the paywall in various areas of the app, because I like to be able to see it sort of clean first and then layer in that side of it. So I've been doing that. Not incredibly interesting to talk about on the show, but hopefully over the next week, I'll be talking about, now I've got a landing page set up and I'm going to hit the button on this and release it. So we'll see.

best live plans and all of that. But let's check in next week and see where I'm

Daniel (49:03.65)
Fingers crossed, I'm gonna hold you to that.

S4e25 (49:06.764)
Yes, yes, absolutely.

Daniel (49:11.384)
All right, then any last thoughts before we call it a

S4e25 (49:19.382)
Oof, put me on the spot there. No. Head empty, nothing inside.

Daniel (49:25.746)
such a nice feeling. I have this visual roadmap on my wall that has just like the words, like words for the different features. And I'm this close to just like being able to strike through the next line, is servers. And that's just such a nice feeling. I'm eyeing, I should bring this to the office, by the way. I'm eyeing this, but I'm eyeing this piece of paper and I'm like, I'm strike this through soon.

S4e25 (49:34.006)
Mm -hmm.

S4e25 (49:53.324)
Yeah, that'll be a, you should capture that for the TikToks. Symbolic striking through.

Daniel (49:59.436)
I but in reality, is a striking through in Affinity Designer and then reprinting it because it looks way cooler this way.

S4e25 (50:11.388)
Daniel. Okay. Well, on that note, on all of those notes, I reckon we should close the show out, dude.

Daniel (50:19.758)
We should, So, listener, listen up. Thanks for listening. Please rate us on iTunes and also like us on YouTube. Please follow us. Please send us emails to contact at waitingforreview .com and also join our Discord. The link is in the show notes. Also, Dave, where can people find

S4e25 (50:42.428)
you can find me actually I always reference Mastodon you can find that in the show notes that's fine but I think this week I'm gonna say you can find me on Instagram find my apps on Instagram and my user there is lightbeamapps .com spelt lightbeamapps and then dot com or one word I shall link that in the show notes

Daniel (51:05.304)
All right.

A different account, Now I wanna have a different account, I don't know. Well, how about this? Find me by listening. Listen to one of my favorite Formula One podcasts called Shift F1. And then just wait until they read out the Patreon sponsors. And when they get to telemetrydeck .com, then you know where to find Telemetry Deck.

S4e25 (51:10.794)
Absolutely.

S4e25 (51:24.236)
Mm -hmm.

S4e25 (51:35.532)
That's very convoluted, but I love it.

Daniel (51:37.592)
It is.

S4e25 (51:41.538)
Yes, yes, we've been over this. You do indeed. It's the rude one that makes me most amused because I'm secretly just 12 inside still mentally.

Daniel (51:42.478)
Because I get all the vowels.

Daniel (51:53.454)
You got a 12 year old inside you, I love that. All right, have a fantastic day. See you soon.

S4e25 (51:57.269)
no, that sounds very wrong. Right.

S4e25 (52:02.42)
Yep, have a good evening Daniel, take care.

Daniel (52:05.272)
Byeeeeee