MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today. I’ve got a great panel with me here today. I've got Eddy, Kyle, Ramses, and Will. And, as usual, I'd like to start off with a story that's not about software development [laughs] and tie it in. 


And I’ve mentioned my dad a couple of times in the podcast. I'm going to be again today. He's an interesting guy. Like, we all think our dads are interesting. Well, I think my dad is interesting, too [laughs]. Most of his career is building, building things, but he's also a tinkerer. He makes his own tools. So, he does woodworking, right? And he makes his own tools for woodworking, and woodworking ends up doing a lot of things. One of the things that woodworking led him into is, you know, making stuff, then he's done some sculpting.


Just in the last month, my mom and dad traveled to Italy and Greece. They hadn't done a whole lot of traveling. Did it there and went and saw some of the sculptures in Europe. And my dad said when he saw Michelangelo's David, it made him cry [laughs]. Because he's done sculpting professionally, you know, he's been building things his whole life. When he saw the craftspersonship, right, that went into that, it was just overwhelming. And, you know, it really affected him, which is pretty cool [laughs]. I haven't had that reaction seeing, well, I've never been there, so I really can't say. But, you know, when you devote yourself to making stuff so much for a long time, it, you know, it really affects you when you see something that's done well. 


Now, we build software [laughs], and you don't see the structure very much. I heard an analogy made by Uncle Bob. Look him up. If you don't know what I'm talking about, he's a luminary in the software community. That's not his full name, but it's what he goes by. He’s one of the signers of the Agile Manifesto. He's a prominent guy. I heard a talk from him once, and he said that in software, we tend to structure our applications around the framework instead of structuring them around what they do, and that idea has really struck me. And he made the comparison to architecture. 


I'm going to start with this, going back to this statue of David. You know, if you're going to make a big marble statue, you are constrained by the chunk of marble you start with. If you don't follow that shape, you're going to have some edges, right, where it does not work. And it drives everything that comes afterwards. You know, everything has to work around the constraints of that stone. And somehow, when we build software, we sometimes don't think that way. If we're writing a Rails app, we think, okay, I've got models, views, controllers. Everything should look like the framework.


And I'm glad we've got the panel we have here today because not all of us are, you know, we're coming from different kinds of frameworks, which is kind of cool. This idea, though, of structure coming from your framework rather than what your code is it's a big one because if you look at your code and say, okay, yeah, this is a Rails application. I've got models, views, and controllers. I have no idea what this application does. I have no idea what the business domains are. I don't know how to find anything [chuckles]. It's all just lumped in there together. 


I'm going to take that further and say that if that's all you see, then there's something probably horribly wrong. Because, in any large application, you've got business logic that has nothing to do with presenting out to the customer. It's probably running in background jobs. It's running...you're probably sending out emails to people. You are, you know, writing reports, whatever the case may be, although you shouldn't write a lot of reports in your app. That's a bad idea. Use an outside tool [laughs]. If you’re writing reports, you're doing it wrong. 


Anyway, but there's all these things other than just, you know, your web app, you know, you probably got a mobile app somewhere. You've got an API. You've got to talk to it, talk to that with. So, if all your business logic is structured around your display code, then you've got a serious problem because it's not reusable.


Going back to what Uncle Bob said, he used the example of not a sculpture but of architecture. He wasn't thinking about that marble you start with, and you get down to the sculpture. He was saying, well, if you look at a floor plan of a building, you know what that building is for. If you look at a school floor plan, you're going to look, and say, okay, here's the gym, here's the, you know, lunchroom, and here's all the classrooms. It's really obvious that it's functionally designed to match that. If you look at an office building and you see your cubicles, you know, and you see your meeting rooms, it's going to make sense. 


You're not going to see every single building look exactly the same because we wouldn't do that because the building should follow the function. We have domains in our code. If you've got something that's, you know, a major component in your code that can make up a...if you've got blogging software, you're probably going to see something around leaving comments, right? And that's a major domain of your application. And if it doesn't look like that, then there's something wrong with how it's structured.


That idea of thinking about where your business logic goes, thinking about how your application is structured, I think, is a potent one. And that's our topic today. Where should you put your business logic? And we're going to talk about this in kind of a framework-agnostic way. Because I think these principles apply universally regardless of what framework we're working in. 


I'm wondering...so, we've got some...so, Will, you're not at Acima. I'm curious what your thoughts. I'm going to start there before we, you know, kind of talk about the Rails side or some of the in-house stuff. I'm also curious what you think, Kyle, and what your thoughts are because you’re looking at this from a DevOps perspective. Where should your business logic go? How do you structure things so that you end up with your business domain and it’s recognizable? And you don't just look like, oh, this is a React app, and that's all it is. I have no idea what else it does. 


WILL: I’d say one thing I am doing a lot of mobile development, you know, that's probably 80% of my work these days. And one thing about mobile app development is you can't roll it out with the speed that you can roll out a web app, right? I can have a web app. I could roll the whole server, I don't know, even a big server I can do, you know, in under an hour, you know, little stuff. You can do it in minutes, if not seconds.


So, one of the things that you really need to be mindful of as a sort of, like, a mobile app developer, is pushing business logic as close to the web server as you possibly can. Sometimes you got to do...I'll do in terms of the data, like, what am I going to render and when, where, why, and how? I always want to push that back onto the server because I can fix it. I can change it. I can get different stuff going. And so, you know, for a mobile app and for a web app, too, I really like to keep that view layer pretty, you know, I mean, in terms of logic, I want to do layout. And I want to do a last line of defense sort of, you know, sanity checking, right?


Because if something, if for whatever reason, I'm the last line of defense to keep something from being screwed up on the screen, that's what you want, right? That's your goal. That's what you want. But, you know, it's hard to hold that line because you're also sort of managing your L3, 4, 5, like, your last layer of caching, so there's performance considerations there. But, I mean, in terms of that, like, that's as far as I want to go. I want, like, the logic of what I'm being presented to be as far down as possible. 


And, I mean, in terms of, you know, business logic, you know, once you start crossing over into web land or server land, I should say, you know, I guess what I'd say is I want to keep the view of what I'm thinking about in terms of logic as small as I possibly can. And maybe I think about it more from the other side because I pick up a lot of, you know, legacy, you know, projects. And so, when I'm trying to make sense of a legacy project, where do I sort of start wrapping my head around it? And I almost always go down to the database schema, like, as low as I possibly can in terms of the data, right? So, I'm looking at database schema. I'm looking at the downstream APIs that are feeding data up to me. It's like, that's how I start to untangle, you know, the logic around some other application.


So, for me, I want to keep it, I think, in general, I want to keep business logic as low as I can get it without duplication. It's like, how do you know, oh, I went too low, you know? Is when I start repeating myself, and I have different subsystems that are doing the same job over and over again. So, I think, you know, broad philosophy-wise. That's the best I've got for you. 


MIKE: So, you've got a layer of business logic that lives close to the data but is not right there. It's a little bit above, so you don't have the same thing replicated everywhere or, you know, in all of your database table interaction probably using ORM. It's not duplicated in all those places. It's a layer above. So, it's reusable as widely as possible. 


WILL: Exactly. Yeah. Yeah. Push it down. Push it down as far as you could push it until you start to repeat yourself, you start duplicating the same logic, the same functions, the same functional units over and over and over. And then, you know, and then you start bundling them into a module so that you can reuse it. And then, you know, you just kind of continue on as you are. 


MIKE: I like that because there's a principle there because it applies both on, you know, you're in your mobile app. You're pushing it as close to the web app as possible which is, the web app is your data layer there, right, that, you know, you're pushing down to there. So, you've got this layer above your data layer for that site that has got your business logic. And then, on the web layer, you've got a similar structure. You want to push down as far as possible, as close to the data layer as you can, so that it's reusable. That sounds like something that can be replicated. Makes some sense. 


So, Kyle, I said I'd ask you as well. So, I'm deliberately going through the non-web app route first before we get into the web app. So, I'm curious what your thoughts are here from the DevOps perspective. Because you write code. You write a lot of code, I'm sure, or at least work with configuration as, you know, as if it was code. What are your thoughts about business logic from a DevOps perspective and where that should go? 


KYLE: I'm trying to think how this would relate in the DevOps world. And what's coming to mind is, specifically here at Acima, we deal with a lot of languages, just to start with. And it's kind of impossible for a small DevOps team to learn every language that all the engineers at Acima are dealing with. And just being able to go in and visualize to see, you know, what is this code doing is very helpful.


So, depending on that structure, which each language is going to have their nuances, if it's going to be in Python versus Ruby, you're going to visualize what's being presented to you in very competing ways just because of the way those languages decided to do things. But kind of like Will is pointing out, no matter what, a language generally has a data schema of some sort. And so, we can kind of see, you know, how is that logic put into a database? You know, how's that being stored, and where does that live? 


And then, the other thing I was thinking about is where we offload the business logic. Where is this offloaded to that's, you know, can it be replicated? Is this in Redis, RDS, those types of tools? You know, does this get put into a data lake where it then gets replicated across? And kind of trying to think about that. 


And then, the other thing I was thinking about, too, is at what point do you start making or putting your business logic into executable functions? And I might mean, you know, Lambdas or the like. When is your code, your app code, just your app, and then the business logic itself gets offloaded onto something else that's kind of executing it? Not necessarily something that is done everywhere. 


I'm just saying, like, when you offload that work where it's constantly repeatable, constantly something that's a small unit that can be thrown away if need be and then rebuilt, and then, like, you're not, you know, necessarily relying on one framework or another, but you're relying on a small piece of code that you can just kind of get rid of and recreate if need be. Those are my initial thoughts on the subject. 


MIKE: Let me dig a little bit more. I know [inaudible 14:28] he uses the analogy a lot of wanting to treat virtual servers, pods, you know, containers like cattle instead of pets. 


KYLE: Yes.


MIKE: [laughs] And it's a vivid image there [laughs], you know, they're disposable, right? This is something that you're not going to have any qualms with getting rid of. 


KYLE: Yes.


MIKE: And spinning back up. And so, I'm thinking you don't want to have little bespoke components that you have to rebuild every time for every new service that comes up. But rather you want to have a lot of reusable [inaudible 15:07] in your infrastructure config and building, rather than having this fragile thing where you have to build every one separately. 


So, I would ask, you know, how do you go about doing that? Because that's logic in building something. You have reusable components for building the infrastructure out. How do you avoid that reuse or avoid the lack of reuse, right? How do you avoid the situation where you end up having to build everything alone? What does that look like in terms of how you structure what you set up? 


KYLE: I'm thinking just everything is config. Everything is config management in that case, especially in the orchestration world. So, you're saving a copy in one place and able to replicate it in several others. And you're taking out those pieces that you aren't relying on that you're able to replicate. So, the server itself, you can replicate that as many times as you want. But the data set, which will probably be the business logic here, has to go into a database that doesn't necessarily...it can have replicas, but you're referencing one data set. And so, everything is kind of going down to that one pipeline. 


And then, you know, depending on what these other services that you are actually utilizing, would depend on whether or not, you know, are you putting anything? Are you storing something in Redis per se? And is that something that can be thrown away? Because if we're not putting business logic in Redis, basically, if we can treat Redis as a cattle, now we can scale Redis out, and we get a lot more speed. However, if we're putting something that's a pet into Redis, well, now we've got to be careful with how we scale out Redis and how we worry about the HA availability of that. So, does that answer your question, what I'm going with?


MIKE: I think it's an interesting answer. You said a couple of things. You said that we need to structure our applications whenever possible to not have special needs. That is, we need to have data, a single source of truth but try to make everything else disposable. And if you're having a caching layer, try to make that disposable. Like, everything disposable if you can, and only have a single source of truth and [laughs] one and only and not any others because that's costly. That's an interesting idea in that it kind of reflects a little bit of what Will was saying that if you go and understand what the data looks like, it can help drive what comes above it.


WILL: Well, but you said...you made an interesting assumption, right? And as I was sort of, like, thinking about mobile apps, right? So, what I'm doing now is effectively just a glorified web app. I have my single source of truth, right, which is a server, and I'm presenting the data, you know, I'm giving data back. I'm sending it to the data...I'm sending it back and forth. 


But that isn't actually how it came up. It came up doing, you know, like, a fitness app. So, specifically, I was doing a line of fitness apps, and they would generate data. They would generate speed data, location data, like, calorie burn data, heart rate data. All this data was like you had a mobile app, which was a lot like cattle in that I have a very capable, fully-featured computer that is reading data and processing it. It's doing its own thing. It's its own process that can do a lot of stuff, right? Easily comparable to one of many server threads generating data. 


But it's also, like, it can do anything. It's just like, oh, I ran under a bridge. We don't have any cell phone service now. I ran out of battery, so this thing's done. The system's going down right now. I ran out of space. This thing just crashed. Oh, you got a virus, and it took down your whole phone. Apple, for whatever reason, didn't like your process anymore and decided to squash it. Anything can happen to these things. They could stop any time at all, for any reason. You know, like, oh, I put it on airplane mode. Oh, I was uploading this big data set, hmm, not no more. We're done now.


And then, you have to sort of catch that, and recover from it, and process it, and get the data back up. It's like, oh, I was sending that thing, but it didn't work out. Oh, and then then I ran out of space. So, I have to delete something. I’m like, they just deleted the whole app, and I had to recreate myself from whatever fragments of state I have available to me. The single source of truth, that's complicated, you know.


And if you're talking about your processes like cattle and not pets, it’s like, oh, well, that job? That job just failed, too big. I remember you know, scaling days at Acima where it was just like, sorry, guys. We made too much money today, and we literally can't count it all. And so, we uploaded these giant CSV files to the bank, like, all of our takings for the day. And I'm like, it will crash [laughs]. It will crash. 


Kyle, you and I, like, I remember you and I were chasing memory leaks in this, you know, giant CSV bank transaction processing thing because we made too much money. We made too much money, you know. And then, we were sort of chasing it down. And so, you have to have sort of like a fault tolerance and checkpointing and, you know, callback steps where it's like, "Okay, did you get that?" "Yes, I got it." "Okay. I'm going to mark this job as finished," you know, and then, oh, is the job idempotent? Can I run the job twice? Oh, maybe not. You don't want to double charge. There's where the pigs get fat, hogs get slaughtered. I'm not trying to double my winnings. That would be bad. And so, when there isn't a single source of truth, that's when things start to get interesting. 


MIKE: So, you’re saying that the structure of things, because we're talking about structure here, becomes challenging when your data is, well, that source of truth, you know, your data is not very reliable, and, well, anything is not reliable. That's interesting. You think about caching, and caching is a mess. Nobody wants to do caching. It's horrible. And it's also the only answer for a whole bunch of problems because of that same problem. There's so many ways, so many ways.


You can't get all the instructions to your processor. You can't get them in there fast enough. So, there is caching in there to get that push down, and you can't upload transactions fast enough sometimes. You're getting a stream of data from an app on the other side. You know, even within the company, right? They're sending you data as things are going on. And you could rely on them and send stuff, you know, make a request, like, a REST request every time. And now you've got a single point of failure, and every time they deploy, you go down, too. So, you've got to have that caching layer. You've got to have redundant data, which affects your structure a lot. 


WILL: Dump it on the queue. You know what I mean? Dump it on your...I misremember the queue flavor, you know, the AWS queue flavor. The SN –


MIKE: SNS is the notification, and then you subscribe to it with SQS.  


WILL: SQS, yeah. SQS or Kafka. Then what happens if you blow out your queue? Uh-oh. Some of these transactions they need to go out and persist to the database. But they don't just, you know, some of them you just persist it out to the database, no problem. But, again, we're talking about people getting paid. You want to commit that atomically. That goes in one time and only one time. And it only commits when all the [inaudible 23:38] are dotted, and ts are crossed, right? And that takes time, especially if you have multiple data centers. 


Again, you don't want to just commit it to one database. You want to make sure everybody got it, and everybody knows they got it. And, like, if you get it, okay. If you get it, okay. And I'm waiting on you [laughter]. Uh-oh, this guy, he's not coming back. So, then I'm dumping my transaction. Transaction is over now. Oh, well, what do we do about that? Now I'm going into the naughty box, where we're going to have to come back to that. Oh, I'm going to retry. How many times do I retry, right? Oh, I've retried it three times. Like, okay, now I'm sending emails. Now pagers are going off, you know [laughter].


KYLE: Well, and it's how do you know when to retry, too? Because you're in the middle of processing a job, and you get a SIGTERM. Okay. Well, did your application handle that gracefully? Is it done? Did it finish? Do we retry now? There's so many cases there. 


WILL: Do we all retry at the same time, right? Like, oh, I just lost a whole...I'm kind of falling behind. And so, then I have a big batch of jobs. Like, I was already not quite doing so good, and then I have a big batch of jobs. They all terminate at the same time. And now I'm just like, [inaudible 25:07] my server because all my retries hid in this giant tsunami, and now we've got pagers going off [laughs] once again. 


MIKE: And that's a general, yeah, and that's not the only place that happens, you know, cache stampedes where you [chuckles] clear your cache, you know, it didn't come through, so I'm just going to wipe it. And a specific problem, I had a different company. We had a content management system, and we were getting a lot of requests, like, a lot, a lot of requests on the order of about a billion a month or something, you know, so a lot of requests coming in.


And, yeah, if you're trying to rebuild, the content for every one of those, oh, you will just die. And [chuckles] so, you have to have layer after layer of guarantees that, you know, when a request comes in, oh, wait, this content is stale. Well, I'm going to go build that in the background. I'm not going to serve you the stale content. And then, when that background content is ready, we're going to stick it in place so that the next person who comes in gets fresh content. But I can't fill up my caches; got finite space. So, I canceled everything forever. And [chuckles] it's messy. And if you ever just take the cache and expect that to rebuild, bad things happen.


KYLE: Well, and it's how much cache do you have, right? Because then it's do you go with a small amount of cache? And, okay, that helps, but a larger amount of cache would be better. But if we lose that large cache, I mean, we're in for even more of a world of hurt. 


MIKE: [laughs] It's interesting we're talking so much about caching and messaging, you know, we're talking about structure and business logic. But we're saying, well, data is messy. And so, inevitably, a lot of what we build is going to be around dealing with imperfect persistence and, you know, imperfect connections, all of the weaknesses in the system because things are going to break. What about just your straight-up reusable business logic? Say you've got code that's going to take an input from your user and split it up into its parts and save it to the database.


Eddy and Ramses, I deliberately went through Will and Kyle first [laughs]. So, I'm going to come up to the web developers and Rails developers specifically. Rails gives you models, views, and controllers. Should you stick your business logic in one of those places? 


EDDY: Just dump everything into your models and then just call it a day [laughs]. You can't go wrong with that. Or also, put everything in your controllers, too, just make them fat, you know, put everything there. 


No [laughs], I think when you started mentioning domain business logic, I started to really write some notes down. And I think what it really comes down to is how large your project is, right? If it's like a simple blog, you know, things like that, I think you're inherently going to have your models and views and everything tightly coupled. And I think that's fine, you know, if it's something that doesn't have much strain in your service or whatever.


But think in terms, like, merchant portal. DHH, I think, is what we call it. It might be still sufficient, but you need to optimize it in such a way so that you're not putting much constraint in your project. So, I think one way to probably think about that is probably just extracting domain objects, things like that. You may want to mock or stub out tests, you know, just so that things are really simple to follow. And you're not waiting for builds and stuff. But still, you're going to run inherently into a problem where your controllers are really coupled with models and stuff. 


And [chuckles] so, I think what we really want to consider is extracting data, right? And I ran into this issue. I ran into this terminology a while back called fat controllers, right? Has anyone heard of that term before? 


MIKE: [inaudible 29:45] as bad thing [laughs]. 


EDDY: Yeah, it's kind of like in dumping ground, right, where you do a bunch of business logic controllers, but that's not really the design for it in MVC, right? Not to mention, I'm a huge subscriber to Singleton, so, like, single responsibility. I think that's really important to also implement into your service. Specifically, if you put stuff in your model, for example, the models, I believe, inherit from ActiveRecord. And they're going to be responsible for a bunch of domain logic and persistence. And I think that's bad, right? So, we want to avoid putting domain or, specifically, business logics into models [inaudible 30:35] controllers, right? Because you're asking for a really bad time. So, something to consider, I think, maybe PORO might be a good alternative, right?


MIKE: So, what you're saying is, don't put it quite in the data layer but have a layer right above that, where you have reusable components as close to your data layer but not quite there. Notice the pattern, which is what Will said should happen in both the mobile app and the web app is that you'd have a layer of business logics. Now, you had talked about having a plain Ruby object. And the specific kind of object really doesn't matter so much. So, you said, well, your business logic should be in its own reusable layer, close to but not quite the same thing as your data layer. 


EDDY: Exactly, yep. You mentioned Uncle Bob's Clean Architecture [laughs], so I looked it up really quick. And something that really caught my eye was something...and I quote, “Software dependency should all point towards higher level business rules. The database, web, and other external interfaces does become the lowest level of implementation details.” And I think that's really key to what we're talking about, and I think it's kind of funny because Rails sort of doesn't follow this design, like, we usually think of Rails as, like, database design is probably considered first. And then, you have ActiveRecord models are created by one by one mappings [laughs]. And I think that sort of happens inevitably when you have a CRUD app, right?


MIKE: Yeah, so there's some pushback from him to rethink about that. Think about what your application looks like, possibly independently of what your database structure looks like, because they should not exactly align. You know, maybe you have a concept that doesn't exactly map to what you have in your database but makes sense as a concept. You can build around it, but it isn't necessarily a thing that maps to an exact table. 


So, we've talked to, you know, somebody doing mobile development, somebody doing DevOps, somebody doing web development. And joining us a little late today, we've got Justin. Thanks for joining us, Justin [laughs].  


JUSTIN: Better late than never. 


MIKE: And I'm curious, so we've been talking about how we should structure an application so that the structure of the application reflects its purpose. And you're doing security, and I'm interested in your perspective on that. And I'm bouncing over you, Ramses, because Eddy spoke up first, but I'm still interested in your opinion if you've got one. I’m going to jump over to Justin because we're getting a wide variety of perspectives here. So, from a security’s perspective, what are your thoughts about the structure of applications? Specifically, where should business logic go and why? 


JUSTIN: So, that's an interesting question because I have never thought of that from a security perspective. Is there a best place for it from a security perspective? If I'm looking at it the way that something is structured, I'm always interested in validating all the input, validating, you know, doing all the checks before any business logic actually gets executed. 


So, you need to be able to do the authentication, do the authorization, you know, validate the inputs. And that's basically it, actually [laughs]. But you got to do those things first, and then you can do business logic. And then, you can, you know, then it gets stored away, you know, whatever happens, happens, the actions, whether it's storing it in a database or calling another third party, or anything like that.


Another thing that's interesting is the inputs and outputs are always interesting, to me, from a security perspective, because, you know, validate the inputs, validate the outputs. And do that for every layer or third party that you're doing, and that makes it a secure environment. I think there are several models that that could fit within, you know, classic MVC, or it should fit within every model, really. But those are my main concerns there from that point of view. 


MIKE: Well, you said something really interesting there. You wanted to have a layer that handles your input, and that's not your business logic layer. 


JUSTIN: Yeah, you got to validate that before it hits the business logic because if you have unvalidated data going through your business, it could hit all the branches, and it may be doing something that you may not want it to.


MIKE: So, from a security perspective, you're advocating for a structure where you have a well-defined layer that does input handling. You talked about Rails, you know, and you mentioned MVC with Rails MVC frameworks. We'll talk about that because we've done a lot with that. Your controller layer is positioned to do that input validation, and I would argue is badly, horribly positioned to do the business logic because it's designed to have that request come in, do the input validation, and, hey, are you logged in [chuckles]? Are you giving me valid data? Are you giving me something huge that I shouldn't be doing anything with? You know, whatever the case is that they're handing off. It's designed to do that. 


But then, if it's doing that, thinking about it, well, that's a separate concern, right? That's a different thing. So, you should be handing it off to do something. So, listeners are not seeing visualization. I'm talking with my hands [laughs]. You've got a layer that speaks to the outside world and handles that. But then the business logic you're pushing back down to a deeper layer that sits in between that interface layer and the database. And this is interesting because we've got a recurring theme again [laughs].


So, Will talked about, well, I'm going to push it as close to the data layer as I can, but not so far that it starts to repeat itself, separating it from the UI concerns, right? Separating from those interface concerns. And he said the same thing about the web application. Well, you're saying the same thing from a security perspective.


And Eddy kind of said similar things from a usability perspective for a developer happiness perspective [laughter]. You don't want to have everything in a place that's not reusable. You want to have it in a reusable place. And all of these are kind of pointing to the same idea that you have this interface layer that handles those interface things. But that's a separate concern from your business logic. Your business logic should go somewhere else. It should go in this deeper layer that's not your data layer but is closer to it. 


WILL: My original thesis, I still believe in strong, right? Which is that it should go, you know, as deep, as close to the data layer as you can get before you have duplication. So, form validation on the UI side actually sucks. It totally sucks from a software architecture thing because there's all kinds of stuff. You're, like, half-ass verifying, and you're half-ass validating, where it's like, I could check maybe a credit card CVC, or, you know, did you even fill this form in? I could do that stuff.


But then I got to do all the other stuff on the data layer, right? So, the credit card form. Like, I got to check this card. Is this card actually...is it valid? Is it even on, right? So, I got to do all this, you know, messy stuff. Like, it doesn't get me out of everything. I'd have to write it twice. And it's actually a giant pain in the ass to revalidate the thing just to be like, I went all the way down to, you know, calling the bank, you know, virtually. And the bank says, “Yo, no fricking way,” right? Then I have to parse that data and then update the UIs. But you want to do it because it's better UX, I mean, it's better user experience, right? You want to hit it as fast as you can. 


But in terms of software architecture, awful, just terrible, terrible. Like, from my perspective, for my benefit only, do it all on the server side, all of it on the server. Because I got to do it that way anyway. So, like, why do it twice? Well, because it's not fun. So, anyway, I mean, it's just, you know, we sacrifice; we bleed --


EDDY: I would counter-argue that you do simple validations on the client side, and then do all of your logic validation in your backend; just saying. But I'm also –-


JUSTIN: But that's what he's saying, right?


WILL: Yeah.


EDDY: Oh, gotcha. I thought you said it in reverse. Okay, never mind. Never mind.


WILL: Yeah, well, yeah, you have to do it twice, right? I'm saying that it sucks to have to do it on the frontend, and then you have to do it all over on the backend. Because I don't trust anything coming from the user, right? And so, purely for my benefit, do it all on the backend, but it's just not good UX, right? Because you've got to wait for the round trip, and then the whole thing, and it's just, you know, it's a giant pain that nobody likes, you know, as a consumer. 


MIKE: So, you just have to kind of forget about the fact that there's another layer that's doing all this somewhere else, and just live in your own layer [laughs]. The other layers don't exist. Then, you don't have to stress about it because you didn't write the validation again. Somebody else did. Except you probably did because you're probably doing both [laughs]. 


WILL: Probably, you're doing both. 


MIKE: [laughs]


WILL: And, at the very least, you have to come back, and you have to parse out the, you know, this is the error, and communicate that, you know, even if it doesn't necessarily fit in your form anymore. Anyway [inaudible [40:55]


KYLE: I was thinking, when Justin was talking, something that we haven't necessarily hit on is actually the testing side of this. And it is one of those things where if you get your business logic too close to your UI level, or if you get it too close to your data level, your matrix on your testing scenarios gets a lot higher, and it gets a lot harder to do. And so, if you're able to isolate your business logic, you can mock and stub a lot of what you need in order to test just your business logic and validate that. And that would be helpful, you know, regardless of whether or not it's QA or security, I would imagine. But that just kind of occurred to me. If we treat it as that separate layer, then we can just focus on that, and curate that, and then curate the controllers and stuff that we're using independently.


EDDY: You know, in the past, I probably would have agreed with you cold-heartedly. But I, more often than not, have wished I had integration testing because it turned out I didn't truly understand, you know, what I was actually touching. So, there is a layer to both.  


KYLE: Well...yes.


MIKE: They can both be true, though. They can both be true. You can have integration tests that do full end-to-end. 


KYLE: Yes. 


MIKE: And they take a hundred times as long. And then, you have your unit tests that don't have to touch the data layer and actually go fast. They're separate concerns that are both important. And if you intermingle those, you'll get in trouble because if you try to do all your tests with integration tests, yeah, they'll never finish, ever [laughter]. And –-


EDDY: Unit tests are fine. You better be damn sure you know all of your branches. Like, that's the only thing. And it's bit us more than once, you know, at Acima [laughs]. And so, all I'm saying is past Eddy would have said, "No, dude, mock, stub, everything. It's fine. You're good." But if you're not actually, like, making calls, you have to be 100% sure, you know, like, you understand how that works, what the response is, everything. 


MIKE: So, you know, I think coming back to the theme we're coalescing around here, from a security perspective, you want to have this middle layer that's had stuff validated before it gets there, right? From a testing perspective, you want to have a business layer that's able to be tested independently that's not all entangled with your data layers. Your test runs way slower [laughs]. And I wasn't kidding with 100x. That's probably actually low [chuckles]. It is way slower to have tests that have to go through [inaudible 43:38], and then they're fragile.


And from a code usability perspective, you don't want to have everything. It all pushes towards...and then, going back to your block of marble, your beautiful block of marble, it gives you some constraints. You've got to have a layer in the middle. You've got to have a layer in the middle. And it's kind of inescapable that that business layer should be somewhere in the middle and not in your interface layer and not in your data layer, but in between. It's important. 


JUSTIN: I hate to bring this up at the end of the podcast, but I'm just wondering, have anyone had experience with any of the cases where they store the business logic in the database as data and then they load it into the middle business layer that way? And if it's stored in data in the database, you can do various things with it. But that dream is always more complicated to implement than people would imagine.


MIKE: I talked to somebody who worked for a company that spent a couple of years building this beautiful system that did that until they went bankrupt because it took so long to build it they never made a penny [laughter]. 


JUSTIN: So, I worked at a place that tried to do that, and it was painful for everyone involved. But I'm just wondering if other people have run into a success story because that's the ideal, right, is you just define it with words in a database, and there you go. 


WILL: I think in terms of the program logic and stuff like that, eh, I think what you can do is sort of...you can have metadata. And one of the things that I've done is I've had sort of metadata around a record, which could be, you know, fairly sophisticated, I mean, really. Like, I'm trying to think if I could characterize this easily. 


So, I'll give you an example, right? So, I used to work on this series of fitness workout apps, right? So, everything was modeled as, you know, exercise. And, you know, it's like, okay, we're going to do chest exercise. We're going to do 20 reps, right? And it would coach you through that in a different way than it would be like, oh, we're going to do 20 minutes on the StairMaster, right, or something like that. 


And so, like, the way it would communicate to you, the way it would log data, stuff like that, you could think of it like a style sheet, like a CSS style sheet. It would interpret things in this way or that way. It would do this thing in this way or that way. And so, it was metadata that did affect the way things are processed, but it wasn't really actual code. I think if you're writing actual code, I mean, because I've seen, you know, kind of PHP-type backends, where they're doing, you know, kind of weird stuff like that. Like, we have databases in databases in databases, you know, where everything is kind of Siamese twinned together. 


And I find that stuff pretty...it's hard to work with. It's hard to think about. Sometimes there are sort of technical reasons, performance reasons, or sort of generality reasons that you would do something like that. But generally speaking, it's more of a means to an end to that goal, you know. I know if you go and you open up Core Data, which is, like, Apple's native mobile data layer, right? It's their version of, you know, SQL, or something like that, right? It shifts with all the iPhones. It's a SQLite database, and every row is the same, right? Every row contains all of the columns for all of the rows, right? And it'll populate, you know, these set of columns just based on the data type that it is, and that's just how they did it. 


And I don't know the internal details around the technical basis for it, but I know some very smart people got paid a lot of money to write a hot, you know, SQLite interpreter for iOS. And I have every reason to believe they had good reasons for choosing what they chose. But if you ever look at it in a SQLite editor, it is just a mess [laughs], you know. 


And so, I don't know, I try to keep it as dumb and as simple and as easy to read as I can, unless I can't, and then I do what I got to do. 


EDDY: Well, I'd like to see that sometime. 


WILL: No, you don't [laughter] [inaudible 48:34] 


EDDY: Hey, just to be clear, I don't want to write it. I don't want to contribute to it, but I'd like to see it. It seems a bunch of spaghetti. 


WILL: Well, I mean, it's exactly what you think. So, if you have, say, think of your random database schema, right? So, you have 10 different data types, right? Every row contains the columns for every data type and some metadata that says, "Oh, this is a product," right? So, it's going to have these columns in it, and they're going to be populated. Or, oh, it's a special sale, and it'll have these, you know, things. It's just a hash. But, I mean, you know, hey, whatever, man. That's the gig. 


And they wrap it up in an only mildly hostile RM, like, you know, APIs, so that you don't have to know about that kind of stuff unless you really, really, really, hate, you know, yourself and what you were going to think about doing that day. Or, and this is what happened to me, something bad happened, and you need to figure out how to undo what you just did. Nobody --


JUSTIN: Hopefully, you can roll back that database update. 


WILL: Exactly, exactly. Rolling back a database update that you screwed up in SQLite is...let's just say anybody who knows how to do that had a real bad day [laughter].


KYLE: [inaudible 49:59]


WILL: All right, well, I don't have anything to add. Should we wrap it up? 


EDDY: Awesome. Oh yeah. This was a great conversation, like always. Super good. You guys are awesome. Thanks everyone for joining. 


WILL: Thank you, gentlemen. Y'all have [inaudible 50:16]


EDDY: Thank you. Another great session of the Acima podcast. Bye, everyone.