Commerce Today Episode 129
===

Joshua: [00:00:00] Hi everybody and welcome to this episode of Commerce Today. So imagine you're starting out your day. You're copying pasting dozens of orders from your website into your ERP, juggling some spreadsheets to update your inventory manually pasting product info from your PIM to your CMS. Uh, does that sound familiar at all?

If so. I'm sorry. But for many brands this really is everyday life. This is how data is being moved from one place to another. You are acting as a sort of data plumber, patching systems together by hand. Another thing that comes to my mind, and I'm not. This old, but I'm old enough to have heard stories and seen old movies of the old school switchboard operators.

So the way the phones used to work, where you would see, um, people sitting in front of a bunch of sockets basically, and when two different people wanted have a phone call, they would actually move the wires to connect those two calls. That's kind of how a lot of y'all, it seems like, are handling, moving data into your e-commerce [00:01:00] platform.

You're out there, you actually have someone. Physically moving those wires, moving that data from one place to another. And obviously it's, it's not much fun for the person that has to do it, but it's also not good for your business. It's air prone, it's costly, apparently manual data entry. Even the best data entry specialists, which it's kind of sad, that's still a, a role these days, um, have around a 1% error rate.

So not that bad, right? But that means that out of every. A hundred, uh, entries, one of them is wrong, and that's probably not a great way to run. Your business might seem small until it's an inventory typo and suddenly you have oversold a key promotion, um, or it angers a big customer or costs you a big B2B sale now.

Even though it's so costly, even though it's kind of risky in a lot of ways 68% of businesses are still doing some sort of manual data process when it comes to their e-commerce site and their ERP and other backend systems. So if this [00:02:00] does sound familiar, you're not the only one doing it, but it's definitely time to fix it.

Also, if you're only updating your inventory once a day because it's a manual process, there's A-A-C-S-V or spreadsheet involved, then you could be overselling products you don't have, or you could have items that are selling, um, well, that you stock out of when you actually have some that just arrived at the warehouse.

So even a 30 minute delay in sinking orders or stock levels could. Cause you to oversell or in some cases basically undersell. It's super interesting. Um, earlier on in creativity's history, when Google product ads were first becoming a thing, we would actually see this whenever a merchant. It didn't have a good inventory integration.

What would happen is everyone else, when a big hot product sold out, their sites would update. It would be marked as out of stock, especially if they're getting a feed from a vendor. But then if there's that one merchant out there that isn't running the feed, isn't processing the feed, it's a manual process to pull that product off their website, they'd sell a ton of [00:03:00] 'em because suddenly they were the only result in Google shopping for that particular product.

That was completely sold out everywhere. So they would end up, uh, back then paying for a bunch of clicks that were for purchases that they couldn't actually fulfill because the item was long sold out straight from the manufacturer. So a lot of reasons to make sure your data is being updated. So this episode is all about fixing your data, plumbing.

~Um, ~it's also going to be the core of a whole chapter in the new e-commerce playbook, uh, book that I'm releasing later this year. So testing out some ideas, testing out, ~um, ~how to share some of these ideas with y'all first on the podcast before it makes it into the book. So want to give you a practical no BS roadmap of how to go from very clunky manual processes to real time integrations.

Actually gonna phase this out in a crawl, walk, run approach. I think what one of the reasons a lot of y'all are still using manual processes and spreadsheets is because you've been told the only alternative is this super [00:04:00] expensive, super complex. All the bells and whistles. Real time integration between all of your systems.

And that's a huge IT project that probably has some IT directors out there already cringing in fear. So you don't have to go from one extreme to the other. There is a middle ground you can look at. So let's actually walk through these steps and look at what this middle ground is. So obviously the, the not even crawling step is that manual process, the, uh, spreadsheet and prey and so.

A lot of y'all are still out there manually wreaking orders, inventory, product, data between systems. I've worked with merchants where different actions on the website would literally trigger. A printer to print out a piece of paper in someone's office and then they would go, usually the poor finance team would take that printout and actually go and manually key it in.

Scary. But there's a lot of people still doing it that way. Or maybe there's somebody that just is downloading yesterday's sales from your stores as well as your website, and [00:05:00] then updating an inventory Excel sheet that you then upload into your website. Um, I've even seen people, you would think if you had a PIM it would be integrated into your e-commerce platform.

That's not always the case. So I've seen people where their product team is actually copying text from their pim, their product information management system into their e-commerce CMS, um, and that's how new products are being launched. Not a great way to do it. So obviously this is consuming employee time that can be spent on, uh, growth projects.

Uh, it introduces errors. It doesn't scale. I had really interesting conversation last week with, uh, another brand I am, uh, consulting with, and their main concern was right now they have so many manual processes that if they try to scale their e-commerce operations, their e-commerce sales, they can't keep up.

They would have to just keep hiring people, and it just, it literally doesn't physically scale. So obviously it's important to get some of these automations, um, in place before you, uh, start to scale up. So. [00:06:00] And again, if you have ever bought something, and then the next day you got an email that the order was canceled because it was actually outta stock, you might have fallen victim to someone that had a manual inventory update process.

See this a lot with the flash sales models where ~E ~either they have a completely manual process or a process that just isn't updating fast enough. So again. Speed is important. So think about how quickly, uh, or how slowly you're getting your new products launched. So whether it's new inventory, restocking an existing product or product data for some new product line that you're adding to your website, it's a completely manual process.

And those things then are sitting in your warehouse for days before they're actually being sold on the website. Your time to market's too slow. Your competitors are gonna be faster than you, and you are not gonna sell as much as they are. So, all right. We hopefully all agree that ~this is, ~this pre crawling stage is not ideal.

You don't want to be moving data between your [00:07:00] e-commerce and other systems completely manually. Good news is you don't have to just like jump straight into that extreme of a full real-time integration. You can crawl, you can do some basic automation with CSVs and quick integration. So start small. In this crawl phase, our goal is to replace the most painful manual tasks with simple, reliable automation.

So this could be batch processes, it could be file-based integration. You're basically moving the burden of copying and pasting data from one of your employees to a computer. And doing that via scheduled imports and exports. Um. It's not real time. And again, as you get even bigger and operate at a larger scale, you could have scalability issues with it, but it's a good start.

So your first friend in this process is the CSV, the Comma Separated Value file. Um, it's basically the simplest way to move data between programs. Um, it is the original [00:08:00] format for spreadsheets, basically. So very simple way that any spreadsheet program and pretty much any e-commerce platform and. Most ERPs can speak CSB, so set a hand keying in orders.

You might export orders from your e-commerce platforms, A CSV, and then import them into your ERP via that CSV file. And that could be an employee actually logging into each of those platforms and doing that. In a lot of cases, you can actually schedule that and just set up a file transfer between the two systems, um, so that it automatically moves from one place to another.

And for a lot of smaller brands, or if you're just getting started and there's not a better way. To easily integrate, say your ERP or your accounting software and your e-commerce platform, ~CSV an ~an automated CSV sync could be all you need for quite some time. It can really take you further than I think a lot of people realize.

~So, I. ~Before you custom build any of this, look for an existing app or extension. Um, there's a lot of existing connectors and extensions out [00:09:00] there. We'll actually see this in a couple of these steps. But at its most basic level, this CSV extension or import or an automatic scheduled ~CSB ~process is something that.

Many developers have already developed an extension or a prebuilt piece of software for, for pretty much every ERP and every e-commerce platform out there. So take a look at that before you try to reinvent the wheel and build a CSV ~sink, um, ~by hand. So if you're not finding something, if say somehow you're using some.

Horrible ERP that doesn't allow you to automate, uh, CSB importer export, or there's not already an extension for it. Lots of times tools like Zapier or make.com will already have an integration, and if you're not familiar with Zapier and make, um, there are two of the different platforms out there that really just specialize in data, plumbing and moving data system A to system B.

So, um, they oftentimes will have an integration into, like, let's say you're [00:10:00] using QuickBooks. You could go and say, okay, I need to pull anytime there is a new product in QuickBooks. Take that data and pipe it over into this e-commerce platform for me. ~Uh. ~So it won't handle really complex ERP processes. It doesn't handle every platform out there, but it can be a good way to crawl with your data plumbing.

So the best advantage to this crawl approach is it's super accessible. You don't necessarily need a developer's, you don't need your IT team. Lots of times this is something that an e-commerce team or an e-commerce manager can set up on their own. It's super low cost. You don't have to go out there and go through a purchasing or bidding or requisition process.

It's usually like Zapier has plans that are like 20 bucks a month. So really a good way to get started. And, uh, on the flip side though. You know, it's not necessarily your best solution or your permanent solution, especially as you scale. So one of the biggest concerns with this approach is latency. Really just a [00:11:00] fancy way to say how long it takes.

So when you're doing CSV based imports and exports, you don't wanna run those like every 30 seconds. Usually those are gonna run once a day. Sometimes once an hour, sometimes once every 15 minutes, and whatever that interval is that both systems support. 'cause you might have an e-commerce platform that says, sure you can do CSV exports once a second for all we care.

But then if your accounting software can't handle that pace, you have to go with the pace of the slower platform. So the CSV import processes typically not gonna see that happen more than every 15 minutes. Some platforms, sadly, it's not more than once a day. So if it's only once a day. Then your inventory, for instance, is only gonna be accurate and up to date once a day.

So you'll be using day old data once you get to like 23 hours after that ~sink ~takes place. ~So, also. ~The error handling on this isn't great and it's ~a, ~an all or nothing approach typically. So if there is one line that is incorrect in that CSV file, the [00:12:00] whole file's gonna fail to process. The whole update isn't gonna go through.

~Um, ~so you have to have really good monitoring on this, and it can be a fairly brittle and error prone approach. Still better than doing it by hand. If you're a ~small, uh, ~small company or if you need ~like ~an immediate fix, you're like, we can't keep doing all this by hand. We have too many errors, crawl might be enough for you.

This step of using these ~CSB ~imports and exports could definitely be enough to get you through. It also could just help you get some quick wins before you kind of start on a larger project for walking or running with your data plumbing. Now next up, let's talk about walking. ~Um, ~this is API, integrations and middleware.

So this is the kind of gold standard that's developed over the past five, 10 years where you're now, um, doing more than just some basic automation, basic movement of files. Um, now you're running at near real time. Um, using different APIs or middleware or [00:13:00] integration platforms, um, to connect systems directly.

So basically we teach these systems to talk to each other directly and continuously instead of waiting for those CSV file exchanges. So. APIs, uh, most modern software is gonna offer an API. There's a few ERP platforms. I'm not gonna name names this time, but y'all know who you are and you know that I'm upset with you.

But there's a few ERP platforms that don't have great APIs. Um, it makes it harder to move your data. In and out of that ERP platform, but most of them do offer an API. It's just basically a doorway for the data. It's instead of exporting a file and importing it somewhere else, your software can just basically communicate with each other and say, Hey, ERP.

Here's a new order. Need you to, um, store it in your system. Or, Hey, website, this inventory level's changed. Please update it. So a lot of, uh, the best e-commerce platforms are API first and open, such as Adobe Commerce and [00:14:00] Magento. Um, they have comprehensive rest and GraphQL APIs outta the box. That basically means that all the modern API languages that your other software pieces might speak.

Adobe and Magento already speak those. So that means basically if you have the right knowhow or the right tools, or you're using the rights, uh, middleware that you've purchased you can very easily communicate between that system and any other system that supports a similar API, and I've already kind of teased this, but you don't have to write code to use APIs.

The application programming interface, they were designed originally for programmers for when a programmer was writing code to connect multiple systems or pull data outta systems. You don't have to be a programmer to use them anymore. There's lots of integration middleware. Um, for instance, there's even, there's a technology called an iPASS, an integration platform as a service.

There's even a great. IPAs called ipass.com. They have a great affordable service that basically is your integration [00:15:00] workbench in the cloud. It has connectors for APIs for many different systems, and it lets you map how the data flows between those APIs through a visual, uh, interface. So. Um, also if you're in the Salesforce ecosystem, MuleSoft, you I'm sure have heard of that and probably spend too much money on that, and that is another integration platform as a service.

So let's actually think about the PIM again. So suppose you have a PIM and that's where all your product data is. If you are doing it right, doing it well, um, your vendors are even submitting their product data into your pim. Your merchandising team is refining it there, transforming it, and then you want that data to be on your e-commerce side as soon as possible.

So you can use ~a, uh, ~an iPASS type platform to connect your PIM and your ~um, ~e-commerce platform. And anytime a change is made in one system, that integration platform will catch it via the API and we'll send it to the other platform. So no more copy and pasting, [00:16:00] no more errors from manual data entry and data movement and no more waiting for the nightly upload.

So huge benefits to this approach, and this is the approach that I see really a majority of the companies using these days. ~Um, ~it's a lot faster than using CSV files. Data can sink in minutes or seconds instead of hours or days. You can even do near real time. Again, this depends on the systems involved.

Some systems can't keep up with this, but. It is possible to do an API integration where an order that is placed on your website shows up in your ERP within 60 seconds. So it could trigger faster fulfillment, faster outreach from a sales rep, whatever makes the most sense for your use case. It's also much more accurate.

~Um, ~there's no manual re-keying. You don't have that issue either where it's these big batch files where if one line is wrong, the whole file fails. ~Um, ~and you can really, ~you can ~handle. Hundreds, thousands, tens of thousands ~of order ~of orders a day through this. So it can [00:17:00] definitely scale. So, yeah, the downside though.

Is, it does require more investment and there is more complexity than ~the ~the crawl stage with CSV files. So typically you're either gonna have to pay someone to build a middleware or you're gonna be buying a SaaS based middleware. Um, so you're gonna have a monthly subscription cost. Um, this can be a few hundred dollars ~to a few hundred dollars a month ~to a few thousand dollars a month, depending on your scale and complexity.

Hmm, that still lots of times can be less expensive than the cost to implement and maintain custom scripts to integrate between APIs. Also you're now deep in the world of software integration, so it can get more complex, especially if you try to do this yourself versus using a prebuilt connector or prebuilt middleware.

A poorly set up integration through the APIs can cause probably just as many, if not more problems as doing everything manually or doing everything via CSV files. [00:18:00] And also it can get a little more technical here. Um, again, if you're using a platform like IPAs, you don't necessarily need a developer, but you do need someone that is savvy to.

Data formats, moving data around, things like that. However, this is a very mature ecosystem, a very mature solution. So there are a lot of people out there that can help you with this. There's agencies like creativity, there's integration specialists and consultants, and typically the ROI is absolutely worth it to go through the effort.

To set this up, and this is where I'll see companies spend a fair amount of time and money to set this up and get it right. But then if you do it right, it's really kind of a set it and forget it approach. All right, so now that you have API driven integrations things are probably moving pretty smoothly.

You can probably stop here, but lots of times people ask me, well, what does a run look like? What does the next step as we scale look like? And that is. Really the [00:19:00] far advanced end of the spectrum. This is real time event driven data flows. This is systems like, um, RabbitMQ and other messaging queues.

Um, it's also known as pub and sub publish and subscribe. So basically whenever something happens in System A, an event is triggered out into this messaging system that says, here's what happened in system A just now. So for example, here's an order that was placed then. Systems B, C, and D can all listen for that event and can process it.

So the event will be sent to those systems. So that could be, it goes to your ERP, it could go to your CRM, wherever it needs to go. But it's not a direct connection like the API integrations we talked about in the walk stage. If you want a completely scalable, the sky is the limit. ~There is no.~

No slowdowns, no bottlenecks, no brittleness. And you want it to be truly real time. This is how you get [00:20:00] there. This is, ~um, ~by setting up that event driven approach. So again, if an order is placed on your e-commerce site ~and an API model your integration, ~your middleware might call the ERPs API to create an order.

~And then that might happen. Well ~first of all, what I see people do sometimes, and this is not how you want to do it, is actually when the user ~places ~submit an order as part of submitting the order, it will send it to the ERP. That means that the page ~actually ~doesn't confirm that the order has been created until the ERP is done.

That slows things down and ~it ~makes your ~ERPA ~performance and scaling bottleneck for your e-commerce site, which is not a good idea. ~It ~that aside, even if you do this the right way, where it kind of does that in the background, it's still, ~um, it's still kind of ~tying your e-commerce site to your ERP in a very direct way ~and ~this event driven model, the second they place the, ~uh, ~order, all ~that the, ~your e-commerce platform has to do is.

Add that order to this message queue or publish that event to this message queue and say, Hey, an order's been placed and there's no waiting to confirm receipt. ~There's ~no ~rate ~waiting to [00:21:00] confirm that it's been processed. All the subscribers of that event, your ERP, your CRM, your shipping system, warehouse system, they will all get it, ~um, ~instantly off that messaging queue.

They can even process it in parallel. So you're not waiting for the ERP to finish before you then go talk to the CRM. They're all each responsible for taking in that data and processing that event. So I already mentioned RabbitMQ. Um, there's also cloud-based services. AWS has a, um, system called SNS or SQS.

~Um, ~Azure has the service bus. Google has pub sub. So lots of, um, kind of proven solutions for this. If you are on an enterprise IT team, then you probably already are using that AWS Azure or Google Service, and now you just need to hook your e-commerce site into that. If you're not on an enterprise IT team and your head is spinning, then you need to decide do you need this level of integration?

And again, like I started the episode with a lot of [00:22:00] companies out there are gonna tell you a lot of, um. Vendors that want to charge you to implement these things are gonna tell you, oh yeah, if you want to integrate, well, the only way to set up this integration is to use a messaging bus. We're gonna spin up, you know, the Azure service bus.

We're gonna spend tens of thousands of dollars in services in configuring all for you, et cetera, et cetera. No, it, there's simpler ways to integrate. There's even simpler ways to set up this event driven messaging approach, but really decide do you need it? And again, ~if you are, ~if you have big, um, visions of scaling your e-commerce site and all of your other technologies either already use a messaging bus or easily support a messaging or service bus.

Yeah, I would look at this, but if you are a smaller business and you don't need real time events that can scale infinitely. Maybe you stick with your integration middleware and the walk phase. Now I will say that some [00:23:00] e-commerce platforms actually will give you this outta the box where you're not having to pay a bunch of money either for the bus or for the messaging system or for the integrations.

And a great example of that is Adobe's app builder. So app builder can serve as ~this, um, ~this layer through what they call the API mesh so that. You're getting this when you purchase Adobe Commerce, so you're already paying for it, or actually you're, you're not, it's included in the Adobe Commerce license and it can greatly simplify, especially with the over a hundred different systems already pre-integrated with App Builder can serve as a great way to integrate at a very high level, very scalable level between your e-commerce platform, your ERP and your other systems.

Now the advantages of this approach it is truly real time and it is incredibly resilient, much so, more so, ~um, ~than any of the other approaches we talked about. So it's the closest thing to instantaneous synchronization. Like you can't necessarily have all of your systems always be [00:24:00] up to the nanosecond in sync with each other, but this approach gets you as close as anything can.

So customers are seeing. Order status updates, inventory, data, et cetera. That is true to the second. You oversell less customers are happier. It also can improve resilience. If your ERP goes down and you have done an API integration in the checkout directly to the ERP, you didn't do that well.

When the ERP goes down, your website might go down. However, when you're doing this event-based setup, if your ERP goes down doesn't make any difference to your website, your website is still publishing those events and when the ERP comes back up, it will process them. So now sounds awesome. But I've already talked about how ~it can be expensive, it can be complex, ~it can be expensive.

It can be the most complex and the most expensive option for your ERP and other data integrations on your e-commerce site. You have to set up the infrastructure. You have to manage that infrastructure, you have to monitor it. And again, this is [00:25:00] where if you use something like Adobe Commerce that has App Builder provided with it.

Okay, now it's not as expensive. Now really your only cost is if you have to write any of the integrations yourself versus if they are already available via the app builder store. So, yeah, that is, and again, you might be wondering, is this right for me? Hopefully I didn't say too many negative things about it.

There's definitely projects I've worked on where this event driven approach to integrations makes total sense. And I'm seeing it come up more and more and make more and more sense on Adobe commerce projects, just again, because of what Adobe's done with giving you app builder. However, if you're outside of that ecosystem, like if you're a Shopify merchant and someone's trying to sell you on this event driven integration, just stop and ask some questions.

Maybe get a second opinion, because typically you're gonna see scaling issues with Shopify itself or with the apps you've bolted onto Shopify before you see scaling issues with a middleware based [00:26:00] integration. So be careful with that and uh, hopefully you find that helpful. So, talked about. Entering data by hand ~talked about ~how to move data via CSV files, via integration levels or layers, and via these wonderful messaging buses and event driven systems.

Really, the main thing I want you to take away from all this is you don't need to be moving your data manually between systems, but you also don't need to panic and assume that you need a super expensive, super complex integration to move that data. Um, I would encourage you sit down and review all of your integrations with your e-commerce site and throughout your e-commerce business, figure out where you are.

Are you crawling, walking, or running? And where do you need to be? And again, this isn't one of those things where you just wanna go all the way to the right, all the way to run, because that's the, the biggest and the best. Figure out where it makes the most sense for you to be, and then start making a plan to get there.

And again. There is no one size fits all recommendation. Um, some merchants may [00:27:00] actually be fine with a CSV based integration. Most of y'all are gonna be fine with the walk stage with an integration ~middle ~middleware. And then, uh, some of y'all really need that event driven set up. So feel free to reach out to me if you're not sure where you sit.

~Um, ~you can find me on LinkedIn. My name is Joshua Warren, and you'll find a creativity gold background behind my, ~uh, ~headshot because there's a lot of Joshua Warrens out there apparently. There's a link there as well as in the show notes on this episode to schedule a free e-commerce brainstorming session with me.

More than happy to walk you through all of your integration woes. And I'll just leave you with a challenge. What is one manual process that your e-commerce team is currently doing? That you can automate in the next quarter. ~Um, ~if you pick one and you have some success with it or start diving into it, leave a comment on this episode or on my LinkedIn profile.

Let me know what it is. And I hope you found this helpful and I'll see all of y'all next week for the next episode at Commerce Today.