The following is a rough transcript which has not been revised by High Signal or the guest. Please check with us before using any quotations from this transcript. Thank you.
===

hugo: [00:00:00] Elena Greywal led Airbnb's data science team from the ground up over seven years, building it into a 200 person org. Afterwards, she moved into political consulting and now teaches data science at Yale, focusing on environmental applications, while also running an ice cream shop. I wanted to hear how she approaches building data systems from scratch.

elena: when I first joined again, it was really early. And so there was a lot of zero to one, which was really fun. it was things like what's going to be the first way that we're going to flag people who might be risky. And that, was an initial very simple heuristic.

hugo: I asked Elena how analytics became a core part of product development at Airbnb from day one.

elena: You want to know, how are people using your product? You want to know. Are there particular customers that are successful? Are there some that are not? How could you help them? And I think analytics is something that should be built in from day one, right? Like you should always [00:01:00] be looking at how people are using your product and what are the blockers and what is going well.

hugo: I've also been really excited to see how the lessons we've learned from scaling data teams in tech are now showing up in places you wouldn't expect, like ice cream shops. Elena's approach to experimentation doesn't stop at Airbnb. She applies those same data driven principles to her brick and mortar business.

elena: when it comes to experimentation,I can share some examples from the ice cream store where it's like the smallest of data and you have an experiment, but I think it's like, what do you mean by an experiment?

Because, there is an experiment that's, really well powered and you're going to get, great statistical significance and, have a perfect control and treatment group. And, the reality is that's hard to achieve actually. And, especially early on,

hugo: I wanted to know how she thinks about the influence of machine learning, whether at Airbnb or even at a smaller scale.

elena: And then for machine learning, again, I think that's really dependent on what is your product, right? Because, for example, on Airbnb, I'd mentioned, fraud [00:02:00] prevention, And trying to predict if someone was a bad actor on the site andthere are.some flags that we could use. But I think that was an example where, pretty early on a model was helpful. And I think it again depends on what are you trying to achieve? And, how much of a gain will you have from building machine learning model versus, having a simple rule or something to just tie you over?

hugo: But even with machine learning, Elena explained that no system works without first asking the right questions.

elena: but again, that's also something that can be determined looking at data. So maybe that analyst could help you to figure that out to see, here's what we're currently seeing in our data and, what is the potential of a machine learning model here. so I think again, that's gonna vary a lot depending on what you're trying to achieve and what the product is.

hugo: When I was in New Haven recently, I mentioned to some people that I knew Elena and they said, how does she manage to sell ice cream even through New England winters? As it turns out, she's got the data to figure that out

And you have significant [00:03:00] seasonality, in selling ice cream as well. But can you tell us a bit about, it's not only the absolute temperature that matters with ice cream, right?

elena: Oh, yes. It's the relative temperature. So what we found is that If it's a sunny day, even if it's cold, after a series of rainy gray days, we will probably be busy. everyone's oh my gosh, it's finally the sun is out. I want ice cream, even if it's, you know, 40 degrees in New Haven, Connecticut.

being mindful of that for sure is really important. And, yeah, noticing those nuances in the trends to say. When do we really think we're going to be busy or not? And how do we prep for that as a result is really important. And then for all of our metrics, I'm always, correcting for seasonality, right

hugo: Finally, I asked Elena how she approaches teaching at Yale and equipping future environmental leaders with essential data skills.

elena: it comes back to our point about really it's around critical thinking and ideas and that's what matters. And I [00:04:00] think a lot of what I've thought about when it comes to teaching people how to work with data is not necessarily how to, implement things exactly perfectly or, Build the right model.

But it's really about how do you ask a good question? And how do you think critically about where you've gotten your data from? What the source was, what it's representing, how complete it is, and then thinking about the cleanliness of the data.

And when I think about data cleanliness, I think aboutcritical attention to detail and how to think about what you've received and not just assume that everything's fine, but to probe the data to ask questions of the data to make sure that it is what you think it is, that it is clean, that there aren't changes we need to make to better represent it.

hugo: Before we jump into the full conversation with Elena, let's just take a quick moment to chat with Duncan Gilchrist at Delphina, who helps make High Signal [00:05:00] possible. 

So I'm here with Duncan, the president of Delfina. What's up Duncan?

duncan: Hey, Hugo,

hugo: So great to see you and thank you once again for making High Signal possible. I thought maybe you could just tell us a bit about what Delfina does.

duncan: Delfina is building AI agents for data science and through the nature of our job, we come across lots of data leaders. And so with the podcast, we're looking to share the high signal.

hugo: Amazing. and I'm just wondering, out of the clips we just showed, was there anything that resonated with you from what Elena discussed?

duncan: Elena's discussion about what an experiment is, in the context of her ice cream shop, I thought was really fascinating. And the piece to me that was most powerful there is thinking through how there's really a spectrum of experimental rigor. When you're a junior data scientist, you tend to have a very rigorous view of what an experiment is.

It needs to have. Large [00:06:00] control and treatment groups. You need outcomes to be stat sick to make any decisions, et cetera. And indeed, a big part of your job is often to fight for rigorous experiments and prevent teams from, prevent teams from claiming wins that aren't real. As a data leader, you often tend to start to have a more nuanced view because you realize there are things that you can't run large, rigorous experiments on that are still super important for the business, and in some ways should be considered an experiment, like a large brand marketing campaign or a major redesign of your product or a big product launch.

And then, as a business owner of her ice cream shop, you realize, in fact, there's another level of experimentation. that you have to understand is binary in nature, and you're constantly trying out new things that are certainly not formally experiments, but are experimentation in kind of the loose sense of the word.

And so I think the this is an interesting [00:07:00] tension that data scientists have to manage and thinking through how to communicate about that spectrum, and, and use it appropriately.

hugo: Absolutely. So with all of that, why don't we jump into the chat? 

Hey there, Elena, and welcome to the show.

elena: Hi, Hugo. It's great to be here. Thanks for having me.

hugo: It's such a pleasure. So I'm so interested in all the things you've done over, over the past decade or so spanning leading data science at Airbnb, consulting in politics, running an ice cream shop and teaching at Yale as well.

So I'm just wondering if you could walk us through bits of your careerand let me know a bit about what motivated each move.

elena: Sure thing. So I guess to start, I grew up in New Haven, Connecticut, and,love growing up here. My mom was a professor at Yale. That was why we were here. And I, was in the public schools and went to private school, got really passionate about public education [00:08:00] and ended up going to Yale and then going to Stanford for grad school to study education policy.

That research ended up being very focused on quantitative education policy research and looking into social networks in schools and building models of social networks. And at the heart of it, I was really interested in how to improve education, how to make it more equitable and, was interested in segregation.

So when students were in the same schools and looked integrated, but their friendships were segregated in addition to that kind of overall school level segregation. And the research was all again doing programming coding. And at Stanford, there are of course, amazing resources for all of those skill sets.

And so took classes and developed the skill set andin what was close to my final year of grad school, was introduced to this concept of data science as a profession, and a friend of mine, [00:09:00] who I had met through playing the card game Bridge, which is a wonderful card game, it was a hobby of mine in grad school, said you could do data science, and I had the thought of like, well, I'm in Silicon Valley, maybe it's worth checking out, and, Why not, you know, looking into it, maybe do a summer internship, that could be a fun thing to do before, going into academia for the rest of my life and being a professor.

And, uh, ended up talking to folks at different companies and went to Airbnb and just really loved it. loved the people I met, loved the Data challenges as well. I was really interested in this concept of people traveling all over the world and staying in each other's homes and learning about other cultures.

I thought it would be really interesting. And I ended up making the move to just take a full time job at Airbnb and leave academia. I think when I did it at the time, it was Oh, I'll try it for six months and see how it goes and then reevaluate. But, ended up loving it. The move was really motivated by something that I thought [00:10:00] would be fun to do, something where I thought there was, learning that I would experience and also where I felt like I could make a difference and that I had a skill set that could make a difference.

And so that was what motivated that move. And it was really out of the box at the time. I think, very few people had left education PhDs to go into data science. It was early on in that, data science transition that, then took off as becoming a pretty common career path.

but,again, it was really those motivators. And I also,like the idea of,being in a place where the research would translate into a change in the site right away, academia tended to be a little bit further from practice, more research separated from practice, and then also the idea of working with lots of people and a team, academia felt a little more lonely.

So those were some of the motivations at that time.and then, that, was a wonderful move and I loved my time at Airbnb. in terms of making the move to political data [00:11:00] consulting after that.maybe more than trying to make a difference in the world and,feeling a little concerned or very concerned about what was happening in politics.

This was 2020 and wanting to do something about it and learn about that world and see if I could help. And that was the motivation to go there. so there was a poll and, I think there was also a little bit of push. I'd been at Airbnb for many years and I felt I wanted a change. I wanted to try something different.

Also moved back to the East Coast, where my family is from, since AirBnB was in San Francisco and so was Stanford. And, gosh, then, did that for a little bit and got interested in kind of more general data science projects for causes that I cared about, which include environmental causes, which led me to teaching the data science class at Yale, which is actually in the school of environment.

So it's a class that's focused on training future environmental leaders in how to use data effectively for decision making. that felt like something that [00:12:00] was really aligned with about and again, another opportunity for learning. I had taught in grad school, but,it'd been a little while, so I was excited to come back to that for the ice cream store.

Now this one is probably the one that feels the most out of left field, but,still related to all the things that I mentioned. I came back to new Haven because I wanted to meet her, my family. And I also felt this is the place I'd come from that had shaped me, that had given me a lot of, wonderful experiences that had helped me in my life.

And I wanted to come back and do something good for my community. And the ice cream store, came to be because it was during the pandemic. And, I think a lot of us were looking for,things to bring us a little joy. And, I was walking around our neighborhood a lot. And, I was like, gosh, there's no place to go for ice cream.

There's no place for dessert. There's a beautiful park nearby that everyone walks around in, and it would be great if, when you went to that park afterwards, you could, get together with friends and have a sweet treat and ice cream. And I also [00:13:00] missed the amazing ice cream culture of San Francisco, where there are fabulous ice cream shops.

And I, wanted to bring a piece of that too and have that in my neighborhood. There actually are great ice cream stores in Connecticut as well. but there wasn't anything in my neighborhood and, the space was available. There was a great storefront that was perfect for this right next to the park.

And I connected with a woman who had started my favorite ice cream store in San Francisco who said she would help me to get it started. And that was really the catalyst for it. And, again, thought it would be something that was fun. I thought it would be something where I'd learn something from it.

And it also seemed like something where I could do some good. And it has been all of those things. it's been really wonderful to see it become a community hub, to see us really leading the way in terms of flavor development for soft serve ice cream. That's our specialty. I think that's been cool, very unique.

not a lot of places specialize in soft serve and those [00:14:00] flavors. And, Yeah, it's been a great learning experience. I've learned a lot about how to manage a brick and mortar business and the types of employees that we need. And it's actually led me to be working on a company with my sister to help with hiring in the hospitality industry.

So who knows what will happen next, but I've always been a big believer in,following, whatever path is unique to you and not, some predetermined path and that, you never know where it'll lead you and staying open to these possibilities is, pretty important.

So I've lived that in my life and, I'm gonna continue doing so and I'm excited to see what's next.

hugo: Wonderful. thank you for sharing that, rich historical context,and the personal journey as, as well. And for everyone listening, if you do find yourself in New Haven, Connecticut, please do check out Elena's. I had the great fortune to be able to eat a delicious ice cream last week at Elena's in New Haven on the day after Thanksgiving.

So everyone should check it out.

elena: [00:15:00] It was great to have you, Hugo. That was really special. So yes, truly open door to anybody who'd like to come.

hugo: Yeah, and I do love that something I heard in your journey. Firstly, the New Haven connection is wonderful, of course, because, I work there and live there for some time in East Rock where you have your ice cream shop. But I do, Can totally relate to one thing is just the difference in time scales of what happens in academia and in industry, particularly with data science, particularly with data functions in, earlier startups as well that the rapid pace of execution and experimentation.

Is electric and exciting and totally overwhelming at many points as well. 

so I would like to go through several stages of your career with you. if you will, do that for me, I am very interested. Can you correct me if I'm wrong? When you joined Airbnb, there wasn't a data function yet. Was there?

elena: There was an analytics team. So there were folks who were called analytics [00:16:00] specialists. There was some title analytics. So there were folks who were looking at data. there wasn't a, there wasn't a large built out data science function though. And that was what that group would then evolve into.

hugo: Fantastic. And what year did you join?

elena: 2012.

hugo: So this is very early in the picture as, as well. I am very interesting you joining when there was an analytics team and then being there for seven years and eventually leading an organization of up to 200 people. Something that I think What we're seeing now is a lot of companies are learning about how to adopt data driven approaches from the data native tech companies that did so over a decade ago.

So I'd love to hear an instructive version of the journey of going from a small team to 200 and the different stages you saw throughout that journey.

elena: Yeah, definitely. when I first joined again, it was really early. And so there was a lot of zero to one, which was really fun. it was things like [00:17:00] what's our what's going to be the first way that we're going to flag people who might be risky. And that, was an initial very simple heuristic.

And then that got built out into a model. And then that eventually became a team of 30 people. And, it was really cool to see how that would evolve over time. Certainly early on, a lot of the work was also instrumentation and making sure we were collecting the right data. I think that's something that is the non sexy part of the job that,becomes really critical is, what are the methods we're using for collecting data?

How do we make sure that those are working well, that we have reliable data, and then we can, use that in an effective way and also building that culture to your point. I think early on we had one data scientist working with a large team potentially and, from, for me, for example, I worked with, and then the identity verification team, right?

And we were basically building out, how do we make sure we know who you are when you log into Airbnb? And again, it was, It was building [00:18:00] relationships more than anything to build trust that, the data was going to be helpful to then, draw important insights from it that then could influence what we were doing next with the team and having those relationships and clear communication so that the data was actually used.

And again, we weren't working in a silo where we were generating great insights or building great models. And then, nothing was making it to the product or nothing was changing decisions. So I think that was really important early on, was like getting in the door with building relationships with leaders and decision makers and also building out that foundation of what is the data we have, how is it stored and organized so that data scientists can use it.

And I think another piece that we also really focused on, which helped build that culture of using data was so that everyone could use it, that there were data sets that were curated that. Pretty much anybody at the company could access and that then they could use that for whatever work they were [00:19:00] doing.

And so that way, we were democratizing the access to data as well throughout that journey. and then, as we grew again, the natural progression was, just to do more specialization, right? So initially it's a lot of 0 to 1 build the first thing.have it be something that works, that's simple, and then, continuing to modify and perfect it as you go along, and continuing to build out the systems to scale.

So a lot of the instrumentation around experimentation, for example, was something that needed to be scaled. The instrumentation around how do we serve machine learning models needed to be built out and scaled. And so those are things that are initially are not important, but then, once you get to scale are really important.

And also, how do we share knowledge too? So when we get to be so big. Initially, you can just talk to everyone one on one. It's easy. You're sitting next to them. you have a team of 200 people. You're not going to know what everyone's doing. So, you know, how do you make sure that if there are learnings or, things that could be shared, that happens.

hugo: Yeah, wonderful. I appreciate all of that. [00:20:00] All of that I am interested. I mean, you spoke early on to the need for Data foundations. And it reminds me of Monica Rogati's AI hierarchy of needs where, of course, people came to her and were like, we need an AI strategy. And she was like, well, do you have your databases set up and do you have the important data?

Then do you have the correct ETL and that type of stuff? And it is an old story in, in, in startup land that. The first data science hire becomes a data engineer for the first 12 to 18 months. So it sounds like getting all of that absolutely sorted, was incredibly integral to your work at Airbnb. 

In terms of democratizing access to data and everything that's possible with it, what is the role of infrastructure in that such as, a platform that allows analysts or data scientists to access the data through to feature stores and shared documentation and knowledge, wikis, whatever it is around.

The plethora of stuff that's available and also what we don't have.

elena: Totally. Yeah. And again, that was something that really evolved and [00:21:00] is continuing to evolve because as we scale, we have More and more data, more and more insights, more and more people that need to access them. And I think initially we had a off site for the entire team. I think maybe it was 2013 and we came up with a set of schemas for core data tables, right?

And that was the first iteration of this, where it was, here's some clear documentation of, a handful of tables that we think are going to be useful for most people at the company, and for the whole data team. And then it really evolved from there. And we had an entire team that was dedicated to tooling for data at the company because of The need and because, there, there were so many important, tools that could be leveraged to help with that process.

And some of them we ended up open sourcing to as well. And some have become companies that people have built out of, out of that work at Airbnb as well. just cause again, having that tooling was so important. So, you know, I think [00:22:00] for any company that wants to have data. be useful and have value from it.

that's going to be a big investment for sure. And again, I think it's something that will continue to evolve because, we are seeing many new innovations and new companies pop up to help with this. And it's really exciting.

hugo: Awesome. And funnily, I'm glad you mentioned that it is a, it is, it has to be a significant investment, but no spoiler alerts, but somewhere I hope to get in this conversation is how you can leverage data when running an ice cream shop. And so there are different degrees of how much we do, whether it's in spreadsheets or databases or whatever it is.

Cause I, I do honestly think that the earlier people think about adopting data driven and evidence driven approaches. Even at an ice cream shop, I really don't mean even at an ice cream shop, but the types of things you can do in terms of thinking about staffing and supply and all of these things, incredibly important.

I am interested and I know part of the answer will be, it very much depends on [00:23:00] the business. If you can give any advice for how to think about, when to adopt basic analytics and then when to start thinking about experimentation, when you need to do machine learning and those types of things and how you thought about that at Airbnb.

elena: Yeah. I think that's a great question. It's something that's a little bit hard to answer in abstract. 

hugo: Maybe you can tell us about what you did at Airbnb and some examples

elena: yeah, sure. So I would say that, in terms of some examples of analytics, like I think of analytics as, you want to find out what is going on, right.

And you want to know, how are people using your product? You want to know. Are there particular customers that are successful? Are there some that are not? How could you help them? And I think analytics is something that should be built in from day one, right? Like you should always be looking at how people are using your product and what are the blockers and what is going well.

And, I think again, that's about [00:24:00] having some instrumentation for your early product, whatever it is, and, making sure that, You do have some data but not overdoing it because you know at the beginning You don't really know exactly what you need to collect all the time The product could be changing really quickly.

And so I think keeping it simple is a really good motto for those early stages where you know, you'll probably bethe face of airbnb the homepage has changed multiple times over the years, right? The different flows have changed over the years. And so you want to invest enough to get the insights from analytics, but, maybe not overdo it.

and I think that's where good analysts can be helpful of what's really needed and what's not. and being parsimonious in that. And then I think also,when it comes to experimentation,I can share some examples from the ice cream store where it's like the smallest of data and you have an experiment, but I think it's like, what do you mean by an experiment?

Because, there is an experiment that's, really well powered and you're going to get, great statistical [00:25:00] significance and, have a perfect control and treatment group. And, the reality is that's hard to achieve actually. And, especially early on, if you don't have.

enough people using your product or, for that particular feature. Sometimes that's, for a subset of your customers. And so you might not have the power, but it would probably still be helpful to run some sort of test and see, even if you don't have the most power, is there anything that you can glean from, having launched it to only part of your users?

And so I think that's something that like Very important to think about and also not wait too long on. But again, you do have to have enough sample size and enough of a tech stack where you could actually implement that, as well. but starting reasonably early so that it does become the culture, right?

Because eventually that is what you want to have built out for pretty much everything that you're launching.and so I think that's like where experimentation come in. And then for machine learning, again, I think [00:26:00] that's really dependent on what is your product, right? Because, for example, on Airbnb, I'd mentioned, fraud prevention, And trying to predict if someone was a bad actor on the site andthere are.some flags that we could use. But I think that was an example where, pretty early on a model was helpful. And I think it again depends on what are you trying to achieve? And, how much of a gain will you have from building machine learning model versus, having a simple rule or something to just tie you over?

but again, that's also something that can be determined looking at data. So maybe that analyst could help you to figure that out to see, here's what we're currently seeing in our data and, what is the potential of a machine learning model here. so I think again, that's gonna vary a lot depending on what you're trying to achieve and what the product is.

hugo: It makes a lot of sense. And I've, I'm excited to later on, talk more about the experimentation and the data you're seeing in your ice cream shop. I am interested in just what it's like to build a data org from [00:27:00] scratch when you were pretty much building the data science organization at Airbnb, what were your Guiding principles.

And . How did you approach structuring the team and just ensuring it's impact scaled with,the growth of Airbnb was pretty serious, right? So how do you ensure your data function can scale with that growth?

elena: Definitely very tricky scaling problem. I'm really accelerated as a result of that fast growth and definitely a lot of learnings there. And I think at the heart early on when I joined Airbnb, one thing that I really valued about the team was that, I mentioned I left academia cause I really didn't want to do research in a bubble.

And I think Airbnb, beginning was really focused on whatever data work we do has to matter to the company and we're not doing it in a silo. We're doing it in a really integrated way with whatever's happening with the business. And yeah, I think that is critical for, having impact and then also knowing how to build out the team because, what you really want to be doing [00:28:00] is, Understanding the business where it's at and how data could be impactful.

And so you have to be really integrated to be able to do that. And I think that helped us a lot to maximize the impact of our data team. And to realize that, at the end of the day, the only thing that mattered was that the company was successful, not that we had built a beautiful. wonderful model.

Like that was not relevant. We needed the company to be successful. And I think that focus is partly why the company was and has been successful, is that really was top of mind. And, then the other piece of that was just really focusing on those hiring processes and, as you're scaling quickly, making sure that you're getting great people and that you're getting People that will also, add to the culture of the team as well.

And that you'll have a team where people can do their best work as a result of that. And so we did a lot of work on our interview process and on making [00:29:00] sure that there was no discrimination in the process and that we ended up, as a result of that, being able to diversify our team quite a bit. So we ended up with about 50 percent women and I think it was about 12 percent underrepresented minorities.

And That was a huge focus that, we wanted to get the very best people. And we knew that if we had discrimination or bias, we really wouldn't be doing that and that would then have a cascading effect of, if we don't have a great team, then we're not going to be having the impact of the company that we need to have.

And,that's the doom cycle for a data team. it all starts with scaling the people more than anything else and that culture.

hugo: So I'm also interested in how you. Set up an org and hire for people who,won't necessarily be a service center and who can be part of the organization and be embedded so that they can actually solve real business problems.

elena: Yeah, I think a lot of that is again, when you're hiring, setting that [00:30:00] expectation that's what the role is and testing communication skills and having interviews with business partners as well, being a part of the interview process. So from the start saying, this is how we operate our team.

And we want to screen to see that you can do that and also set the expectation that you can do that. And I think that's a lot of also. Yeah, getting people that can talk to others, to be honest, like that's a really important part of having influence is, again, doing your work, but then being able to share it and making sure that we can translate whatever work we're doing, to having an impact.

And we actually ended up having someone who focused on communications coaching on our data science team for this reason, because we were like, this is so important and not everyone has this skill set. So we want to help them to develop it.

hugo: yeah, that makes sense. And structurally, it's coming out, you can break down the way we structure data orgs in a variety of ways, but one way to slice it is. You have, a centralized approach where you have a data team that things come in, things [00:31:00] come out, you have an embedded model where you have data science team, data scientists embedded in different, like someone on the marketing team, for example, right?

Someone on a product team and. of course, you know this, I'm saying this for the audience as well. and a hybrid model where you'll have people embedded, but, or I'm sorry, in a centralized data science thing, but with dotted lines and embedded. So which of those have you found, or another one works best to be able to have people who deliver impact as

elena: Yeah. I'm going to give a answer that's probably not going to help people in terms of a silver bullet. But again, I've seen it work in different ways, there's really not just like one way. And in some ways it's who are the people that you have, right? And what are the problems you have?

And it might be that a mix of the two is helpful, right? for example, I think at one point we had. Some really important data science questions that, weren't necessarily fitting into one of the teams that existed, and it was actually helpful to have, a handful of people who were not embedded at all, even though most of the team was, right?

And so it was [00:32:00] like, for this particular thing that we need to do that will help the company, it would be better to have, this handful of data scientists working on their own in what you might think of as a silo, but for a specific reason. And so I think again, it's what is the stage of your company, where are you at?

Who are the people on your team? How do they interface with the rest of the organization? And that might inform what's really the best setup. I think at the end of the day, In order to have impact, you have to have context on what's happening at the company. again, if you are too siloed, that may not be happening.

So that's a risk. and at the same time, you also want to have,practices that are excellent. And a lot of times that is facilitated by, data scientists working with other data scientists. So potentially having some aspects that are not embedded, becauseif you're a sole data scientist working on your own, embedded on a team, it may be harder for you to develop your data science [00:33:00] skills and again, to have that kind of practice that's excellent in your data science function.

So again, I think having mixes of the two is always a good idea. And I'm hesitant when people are like, this is the only way. I'm like, is it? probably not. again, I'm very much focused on like the end result. If the end result is working and your team is doing well at the company and having an impact, keep doing that.

great. If it's not, you may want to change your structure. that could be a factor of why you're not being successful.

hugo: absolutely. And I appreciate there's no silver bullet. Are there any heuristics that you had in mind or that would help people figure this out? I like the heuristic of make sure you drive business impact,

Yeah. I think that's the main one. But dream of that.

elena: maybe size of team is a part of this too. I think, when the team is really small relative to the rest of the company, it's pretty hard to be embedded. that's gonna be a hard one to do. you're gonna have one person matched with 50 people, maybe, and that's gonna be very difficult.

depending on that size of team and kind of ratio to the rest of the company and data [00:34:00] needs, it may make more sense to have a centralized team that's You know, potentially taking tickets just because not tickets per se, having some sort of queue of work that isn't necessarily embedded because you really just don't have the bandwidth to be embedded, and that will be the most impactful way for you to do your work.

So I think that's one heuristic. I think the other piece of this is, the seniority of your team, right? I think more junior people tend to need more And so you don't want a junior person off on their own without help, and that person might benefit from a little more centralization and less embeddedness.

Whereas if you have people who are good to go on their own and, working great with their team embedded, then that's awesome. keep that going. Yeah,

hugo: am. I also am interested in what you learned about leadership as Airbnb scaled because from the way you described your story anyway, like a lot of people, it [00:35:00] seems like you weren't trained to be a manager necessarily, or anything along those lines, but then you were leading an organization of 200 people.

So I'm just wondering what you learned about leadership and scaling teams. That you think data leaders today might find and find useful. 

elena: I will say that. I didn't necessarily have experience being a manager prior to Airbnb, but I did invest in learning about management quite a bit. And I had many great mentors. There were kind of leadership programs that I participated in that were around, training. And so that was really helpful.

and then I think, it's about building trust with your team and building relationships and knowing the company well to be able to understand where the team can have the most impact. And a lot of it in terms of leadership is that trust, both trust with the company and trust with your team.

And, I've thought a lot about that. And I think this is a leadership quality that isn't just for data science, but for anybody, I think, at the end of the day, you're a data science [00:36:00] leader, but a data science team is a team of human beings, like they're people. and the leadership skills are.

Around how do you motivate people? How do you get the best from people? How do you inspire people to do again their best work? And I think a lot of that comes down to trust and that they trust you to have their back and that you have a community of trust. And I'll share just like one anecdote, around this and this kind of relates to again, how do you have a great team?

And that's what a leader is responsible for is a great team.we were doing so much hiring, and so I mentioned, our hiring process being really important. And a big thing that I was doing was basically continuously meeting with data scientists, with data science leaders to find great talent, right?

Constantly having lunches, just, networking. and I remember there was one person I had a lunch with and afterwards a woman on my team came up to me and was like, We can't [00:37:00] hire that person. And I was like, what do you mean? what? and she was like, I don't want to say more, but we can't hire them.

don't bring them on the team. And so I went back to her and I was like, look, like that's really not enough information. Like I, I do need to know what is the reason behind this. And she said, let me tell you privately. And so we, went into a separate room and she shared with me that this was someone that she had known.

And that, on one occasion he had physically attacked her and she hadn't shared that with anybody else and she would feel really uncomfortable with this person joining our team. And I was like, certainly we will not be bringing this person onto our team. Like that is a good reason. Like we do not need to move forward with any further steps with this person.

And I think the story reflects that. there was that trust, right? That, that there was trust that, this person could open up to me about this. And that at the end of the day, it was about protecting our team, [00:38:00] making sure everyone was safe. They were able to do great work and that we didn't have any bad apples on the team that we're going to really hurt the culture of the team.

And I think that's the aspect of leadership that, people don't really talk about very much, but you'd be surprised, like that comes up. And. that's the kind of decisions that I think make for a great team and are the types of principles that I found effective, be effective. 

hugo: so now I'd like to move onto thinking about experimentation and then diving into the other things you've been working on since Airbnb, especially experimentation across context. So it's been a thread running through your work from Airbnb to political consulting, to your ice cream shop. So I'm wondering what principles have stayed the same of experimentation, And what shifts depending on the context.

And I actually want to say one more thing. I do think we talk about.experimentation in a data driven sense, being a relatively new phenomenon. And in [00:39:00] some ways that's right, especially with all the technology we have to back it up. But I do think a lot of business leaders for centuries have been wonderful at experimentation.

It may not be the same, what we envisage is experimentation. But to run a business successfully, you've actually had to be very good at experimentation, at least evidence driven experimentation, then doubling down on things that are working, right?

elena: Totally. Yes. And I think you're right. There has been a long history of experimentation in business, even if it hasn't looked exactly the same as this.

hugo: So what has stayed the same and what has changed for you?

elena: So I would say that, the thing that has stayed the same is, And understanding that we may not be right and that, we may not know what is the right answer and that kind of attitude means that we should be testing what we're doing and seeing if it's working or not, instead of just going with it and never looking back.

And I think that's really important. And in many [00:40:00] industries, I do think that people will become,a little bit cocky and overconfident that,they've got the answer and that, there's no reason to test it. And so I think having the attitude of Yes, we should be testing because we could be wrong is the thing that pervades pretty much any setting in terms of making sure that we're testing effectively.

jumping from Airbnb. Airbnb was, of course, amazing for testing because we had all this data and we had a product that we could turn on and off for different Parts of our customer base and see, did it work? Did it not? And to be clear, it was also quite complicated in many instances and hard to get right.

but certainly,we had metrics that we cared about that we could see the outcome, right? And then coming to the political data sphere and doing political data work, the end outcome that you care about is at the end of the day, the election result, which is a one time thing. That's gonna happen, and you can be running all these experiments before that.

But ultimately, it's very hard to replicate that exact [00:41:00] outcome of that vote on that final day, right? And so much is changing. Between the time you're running your test and that day. And so I think that's a big challenge for political data and experimentation when it comes to things like Is this tactic working or not?

And so you have to look for Upstream signals that you know might indicate predicting that future outcome to if we're Knocking on doors and we're having people tell us they're going to vote for them for the candidate. We're supporting then You That's, potentially predictive of who's gonna be the person they eventually vote for, right?

But again, it's not that end outcome in the same way that, on Airbnb, making a booking with a host might be where it's clearer. So I think that's a big challenge. And then also the size of the data is a big challenge. Of course, again, coming back to that, you might not have statistical significance, but it's still worth trying and you get some signal.

And, for the ice cream store, we [00:42:00] are doing tons of testing in the sense that we're just trying things out and seeing how they go, right? It's not an A/B test, but it's a Every Wednesday, we have what's called Wednesday Test Kitchen, and Wednesday Test Kitchen is a day where our team can come up with a flavor or a topping, anything they want, and we're going to test it, and I will not say no.

that's the policy. whatever it is, go for it. And,then we'll see how do customers react, right? And we can see What are the sales data for that day, right? We can definitely see differences there, and we can see what do people tell us. We get requests. So you see the revealed behavior and actual behavior of what they're purchasing and then the kind of level of enthusiasm as well.

And so those are some of the data points that we look at for that Wednesday test kitchen. We've also done blind taste testing as well. So that's another fun level of experimentation. again, just getting [00:43:00] together, 15, 20 people and saying, here are some different flavors, rate them privately and let's see the results.

And, sometimes you have 15 people, but 14 out of 15 say they love one versus the other. And that's enough. you're like, cool. I think we can conclude this one is better. And we did that quite a lot early on, especially for deciding what some of our kind of core toppings would be and core flavors.

And I think that's a great example where, it's not a setting with big data, we don't have an AB testing framework, but you can do a test and you can see what's working.

hugo: That's super cool. And I do want to double down on, on your ice cream story now, because to your point, you can run tests. You don't run a B tests per se. And you, You don't run a tests to make sure your instrumentation and telemetry is working right. You have used data to make business decisions.

And as you know, um, my friends who I stayed with last week in New Haven, I told them I knew Elena and they're like, Elena's ice cream. [00:44:00] And I said, yeah. and they said, How does she do it? And I said, what do you mean? And they said, she's got lines going around the block in winter. She can sell ice cream in the depths of winter.

so I'm wondering using that kind of as a,a launch point. Can you share some examples of how you've applied data driven approaches to thinking about staffing flavors, or even when to open and these types of things, and when people will be interested in ice cream.

elena: Yeah. it's funny, Yeah, these are all things, questions that you do have data on and are actually not too hard to use data to inform. in terms of the question of what we have, like I said, we look at our sales data and we look at enthusiasm and we do these A B tests. So that really helps us to know what is good and what isn't and again, very, very much, open to the idea that like people have different tastes and what I think might be best might not be.

And so I think that's like really important for going at [00:45:00] it and figuring out what to offer. And I don't know that's always the case at every business. So I think that has really helped us to be successful. yeah. Listening to people through what they tell us and also through what they're purchasing when it comes to staffing, we look very closely at metrics like how many customers we've served per hour and that helps us to inform both how efficient we're being and also How many people do we need to staff at a given time?

And we have been extremely busy We've had lines out the door and you know improving that efficiency metric has been really important and we found things like You know, a lot of people were writing to us. Why don't you have more staff? Like you're so busy, like you need to have more staff.

And so we experimented with adding more staff behind the counter. And we found that actually was worse because we don't have a large space. And it became actually more inefficient because people were running into each other. They were waiting for the other person to finish with the ice cream machine.

And. It just, it actually wasn't helpful to have more people serving behind the counter. But [00:46:00] the thing that was helpful was that we would then have an extra person who was there solely for restocking. Because if the people serving the ice cream and taking orders from customers got distracted or, taken off the lines is how we refer to it by having a restock, then that really slowed down the service.

And again, this is something that you can see in the data of, like, how many customers per hour are we serving? and then, in terms of when to open, again, very clear in the data. you can see when we're busy, when we're not. And I think, also, the experimentation mindset comes in. initially, I was like, let's just be open more hours and see what happens, right?

Because I don't know which hours will be the busiest. And there's a value in trying out more hours and seeing which ones really are the busiest. And what we found was that, weekday in the morning, it was just way, way too empty. people were not getting ice cream on a weekday, at noon.

And a lot of ice cream [00:47:00] stores are actually open at noon, and I'm always very curious about that, because I'm like, looking at our data, noon to one, everyone's eating lunch. Nobody's getting ice cream.

hugo: open it through.

elena: Yeah.

hugo: why you open at 3 PM.

elena: So we were like, let's open at 3 p. m. That is when we see enough people coming through to make it worth being open.

And, in the evening, the same thing, we were like, okay, actually after 9 30, it really dies down. we're in a residential neighborhood. We're not in a downtown area with lots of clubs and people partying at night. And so it doesn't make sense for us to be open past 9 30 for that reason.

So it's stuff like that where I think it's just being very disciplined about, we're not going to be tied to what other ice cream stores are doing or what people have done before. We're going to be looking at our numbers. And we're also looking at how that changes throughout the seasons because the seasonality for ice cream is very real.

And we want to make sure that we are responsive to how the business fluctuates as a result.

hugo: And this is one aspect, which I think maybe there's [00:48:00] similarities to Airbnb, like in a very general sense, but seasonality, I presume is very important in thinking about. the availability of properties during Airbnb and incentivizing people to make their properties available and that type of stuff.

And you have significant seasonality, in selling ice cream as well. But can you tell us a bit about, it's not only the absolute temperature that matters with ice cream, right?

elena: Oh, yes. It's the relative temperature. So what we found is that If it's a sunny day, even if it's cold, after a series of rainy gray days, we will probably be busy. everyone's oh my gosh, it's finally the sun is out. I want ice cream, even if it's, you know, 40 degrees in New Haven, Connecticut.

being mindful of that for sure is really important. And, yeah, noticing those nuances in the trends to say. When do we really think we're going to be busy or not? And how do we prep for that as a result is really important. And then for all of our metrics, I'm always, correcting for seasonality, right?

there's no,raw [00:49:00] data that's shown of, this is the best seller. It's always corrected for with seasonality. So vanilla is our base. People order vanilla at pretty regular rates. So I'm always like, if we have a special flavor, in the winter, the sales are going to be less than if we have it in the summer, but we can look at the ratio to vanilla as a way to be like, okay, How is it relative to the other flavors as a metric?

hugo: Fascinating. And when you mentioned the importance of the Delta, the differential with the weather the day before, when we spoke a few weeks ago, I told you like it was. I'm in Brooklyn at the moment. I live in Australia again these days, but when I used to live in New York, I used to notice it could be freezing and you'd have people eating ice cream.

If it was, several degrees colder, five degrees Celsius colder the day before or something like that. And you validated that for me. So that's a fantastic insight. And to think that actually. Even thinking about forecasts and how you need your supply chain in place, make sure you have everything, you've got your staffing in place, all of these things, to be able to serve higher demand.

It's [00:50:00] fascinating.

elena: Definitely. And then there's events. And this is something we saw at Airbnb too, right? Where you're like, the Super Bowl is coming to town, Airbnb's are going to be sold out. And it's the same thing in New Haven. We have to be aware of what's happening in the city. And are there going to be a lot of people in town?

Yale graduation weekend, we were the busiest we've ever been because all of a sudden there are so many people in town. And so we have to plan for that.

hugo: The other thing that I find fascinating about what you're doing now is you're very much on the ground in the shop as well and figuring out with your staff, all of these decisions, which, which I think is wonderful. And it just reminded me I think it was McKinsey went in, it was McKinsey or one of someone like McKinsey going to help McDonald's, I think, optimize how like burger patties traveled through like the back rooms and got on the fryer and that type of stuff.

And it costs McDonald's, X hundred thousand [00:51:00] dollars.McKinsey made all these suggestions in the end and then the people on the ground were like, Oh, we could have told you that. Yeah.if you'd come and asked us what we need to make these patties go from there to there more quickly, you could, we could have told you.

and so I think that just speaks to the idea of being on the ground and knowing exactly what's happening. and speaking with everyone there.

elena: Yeah, and I think, again, that's a leadership lesson that spans whatever industry, right? you have to know what's actually happening. And especially, for the ice cream store with a team of about 15, 20 people, it's not that big. I mean,you really do wanna be on the ground there to know what's going on and ha and have a great team that has a culture of trust where people can tell you if something's going wrong.

And, we have had so many great suggestions come from our team and that is certainly critical to the success of the business. and again, the culture on the team. And I think, you talked about the lines out the door. so many times I hear [00:52:00] from people that it's a really friendly, fun line and that they love the experience.

And part of what they've told me is that. they feel like my staff, the staff is having a great time 

hugo: and that it's just like they love the kind of positive energy. And so it's okay that they're waiting in line. Like they don't mind. It's like a pleasant experience. it's like this happy occasion.

elena: And I think, yeah, like having a culture where people feel like they can speak up about something that isn't, And going well and, help overall is really great. And it leads to, that feeling of, happiness and joy in your job that then translates to all your customers

hugo: Absolutely. And you mentioned, there's a, you experiment with flavors and that's driven by your staff as well. Not you. Correct.

elena: a hundred percent. Yeah. And I think, again, this is a great leadership lesson because, I,I always say again, your goal as a leader is not to know everything is to help your team to be successful. And, I. [00:53:00] Pretty early on was like, I'm not going to be the best at figuring out flavors.

And if I'm the one who's saying, Oh, we should do this flavor or not. it's going to have too much sway, right? Like it's I don't, and I don't want to have that sway. I want us to be able to experiment and test things out. AndI don't want to be listened to too much. Like I was like, you guys go for it.

And I can weigh in after, but I don't want to be,I want there to be this creativity that's completely unstifled, that people can just try anything and not be worried about, there being something that goes wrong. And,I think, that has been really great, and again, we've had just amazing flavors.

We've had one team member in particular who's really stepped up and taken lead on the flavor experimentation, and she's done an incredible job. And, again, it's like,I could never have come up with those flavors that she's come up with, right? And she could never have come up with those flavors if I had been an owner who was like, I want to have mint chocolate chip or I want to [00:54:00] have, XYZ.

So there, there has to be that. I think that culture of just letting people. try things out. And, I remember there was one flavor she tried that was, Christmas tree. It was last year. And she, boiled the spruce needles to create a syrup out of the spruce needles. And I think, in my mind when she was doing it, I was like, this seems a little crazy.

is this going to be, is this going to be okay? what's going to happen? And then there were people who were like, this is the best thing I've ever tasted. I was like, okay, like again, you just, you never know what can happen from people trying things out. And it's like having, having a culture where that's really the norm and encouraged is just so wonderful.

And I think it's so fun for everyone involved because creativity is fun. Like it's it's exciting to see what people come up with and to try things out.

hugo: Without a doubt. And that, to bring this back to like data teams as well, I think I'm going to have, Eric Colson on the podcast [00:55:00] soon, who was, VP of data science and ML at Netflix and then chief algorithms officer at stitch fix for the best part. Something we're going to talk about. He actually published an essay recently about we should, for the most part, we should hire data scientists for their ideas, not for their models and execution and not treat them like a service center.

and part of, he gave an example that, at one of the orgs he worked at, They got customers, to do a survey, and then they segmented, the customers based on the survey and what their marketing team, something along these lines, thought the segmentation could look like. And they didn't see any difference of behavior across the segments.

and it just so happened that a data scientist was in the room at that point. And they were like, Oh, why don't I take this data? And do a segmentation, some sort of cluster analysis, unsupervised learning based around it. And they came back with totally different segments, which were then interpretable and made sense.

And then they were able to market different products to these different groups, and they were able to do [00:56:00] that. Not because they were treated as a service center, but because they were in the room with the people with business needs, able to think creatively about. So that isn't too far from someone making.

Having the freedom to make Christmas tree flavored ice cream, right?

elena: totally. Yeah. I think that's exactly right. It's, having that ability to think creatively and come up with new ideas and that is where you just see so much value and have great results as a result.

hugo: I love it. And I do think it also speaks to, it speaks to giving your staff freedom, but that also comes back to another point that you indexed on quite heavily earlier, which is building trust and how you can have freedom. We can all have freedom, but to work with people and have freedom, we do need a relationship of trust and knowledge there.

elena: definitely. that Christmas tree was because we had a great relationship and I trusted her to try it. And, she was comfortable enough to take that risk too. And, not be worried. Oh, maybe someone won't [00:57:00] like it. But to know that if they didn't like it, I had her back.

And, I think that's something that, I tell our team too, where, sometimes something doesn't work and I'm like, excellent. Great. I think this is a really good sign because if everything we were doing worked, we would, it would mean we were not taking chances.

hugo: absolutely. Yeah. Negative results are a beautiful thing, but we need to, yeah, we need to invest in that knowledge as well and be continuously learning. I am, I'm so interested in what you're up to teaching at Yale now as well, Elena. you're equipping the next generation of data scientists or people thinking centric way.

to do the work. So I'm just wondering how you decide what to teach and what data skills do you think the next generation of people working with data will need?

elena: Yeah, I think that's a great question. And it comes back to our point about really it's around critical thinking and ideas and that's what matters. And I think a lot of what I've thought about when it comes to teaching [00:58:00] people how to work with data is not necessarily how to, implement things exactly perfectly or, Build the right model.

To be honest, I think we've talked a little bit about this, but,there are some great tools that help people to do that now, and it's a lot easier. But it's really about how do you ask a good question? And how do you think critically about where you've gotten your data from? What the source was, what it's representing, how complete it is, and then thinking about the cleanliness of the data.

And when I think about data cleanliness, I think aboutcritical attention to detail and how to think about what you've received and not just assume that everything's fine, but to probe the data to ask questions of the data to make sure that it is what you think it is, that it is clean, that there aren't changes we need to make to better represent it.

And, in my class that can show up as something simple like. fixing a string to be the right format, [00:59:00] right? But at the end of the day, again, it's about that attitude of I've received something that I'm going to be using that's data to make a decision, but let me think about it critically.

Let me not just take it at face value and, let me make sure that it is what I actually need to answer this question.

hugo: Very cool. I am also interested in whether generative AI has impacted the way you teach or what you teach or how you teach.

elena: Yeah, this is something that I'm really interested in, and I would love to hear from maybe some of your listeners about some advice for this. we did start seeing, many students using ChatGPT, and, my attitude has always been that the goal of the class is to teach you how to work with data with all the available tools at your fingertips.

that means, open computer, go ahead and Google things. Go ahead and use chat GPT. The challenge that I found is that in order to use anything [01:00:00] effectively, you need to know enough to know whether you're getting the right answer back. And I think that's something that has been tricky, where a lot of students might just use chat GPT to write the code and then, the answer they got really isn't good, but they don't necessarily know enough to evaluate it.

And so it's about, really helping students to get that foundational knowledge so that they can know what question to ask. generative AI, right? What and how to assess what follow ups task is the answer I'm getting reasonable? And if not, what, where else can I turn to try and figure out how to get this right answer?

And so, you know, I, I think that's like kind of a whole set of challenges and it's something that I've really been thinking about in terms of like how much to allow students to use those tools versus not because the goal is to not use them. The goal is to use them. But the goal is to use all the tools that you have effectively to get to do [01:01:00] great data work, right?

So I think this is something that i'm also figuring out and something that we'll be working on for the upcoming semester and i'm definitely open to suggestions and ideas of Yeah, like how do you Set students up well to use these tools effectively and what are some tips to do that?

hugo: I love that so much. And I think, something you've almost stated explicitly is we need to understand the technology we're using to use it effectively. And most software we like word processors or Excel spreadsheets, or even Grammarly and spellchecks. Like we know what they do. we know, may not understand all the like compute, like machine code and stuff that allows them to run, but we know what they do, what they're good at, what they can't do.

And I do. I think we are almost in a public awareness crisis with respect to large language models, and I don't think enough people realize that they're optimized to seem helpful as opposed to being optimized for accuracy, or for giving, ground truth in information. what that means is that they're [01:02:00] eminent people pleasers and occasional gaslighters as well, to be honest, they'll be like, Oh, no, but this is really what I said.

on top of that, I think, so for example, ifit's really important to know that they're often quite good with stuff that they're got a lot of training data on. So if I've got questions about like pure Python code or something, It will be very good, right? Whereas if it's about current, if it's about, earlier this year, actually, open AI changed its API, but chat GPT is cutoff date was before the change.

So chat GPT itself didn't know the new open AI API. So it constantly gave incorrect code for the open AI API. And that seems very counterintuitive, but if you know, It's trained on this data. It hasn't been updated since then you can reset your own expectations around what it's good at and whatnot as well.

I think.

elena: Totally. No, I think that's exactly right. [01:03:00] So yeah, that understanding. And again, that comes down to just critical thinking skills, like not taking what you get at face value, but understanding how to probe it to know what's really going on.

hugo: yeah. And I was in a car with a family member driving who I won't mention their name or my relationship to them, but, they nearly went down a one way street once the wrong way. I said, what are you doing? And he said, Google told me to go there. and I was like, okay, I get that, but just look at the sign as well.

And look at the straight. So I think once again, some sense of being critical thinking and sensible with respect to technology will be increasingly important, particularly as we rely on it so much.

elena: Totally. Yeah.

hugo: thank you for sharing everything that's happening at Yale. That's all super, super exciting.

elena: And I'm kind of jealous. I mean, when I was doing my postdoc at Yale, I got to, pop in on from two lectures from some of the people I've admired for so long. So for Yale students to have access to all of your wonderful education with [01:04:00] all of your experiences is incredible as well. Hugo.

hugo: in terms of wrapping up, I would, I'm just, I know it's been a while since you've worked as a data leader, but you have,you've learned so many different lessons.

So I'm just wondering, looking ahead, as data continues to increasingly shape, industries, society, what advice would you give to working and aspiring data leaders, about staying impactful and relevant in such a fast changing field?

elena: Yeah, I think that the thing that I always come back to is just what are you trying to achieve? What's the problem you're trying to solve? And to be a great data scientist and a great data leader, I think you need to make sure that you understand that context and that's, Always going to be valuable to understand.

And the models, the tools, those are going to keep changing. And so to stay nimble, To recognize that, there's going to be continued automation [01:05:00] and that's a beautiful thing and that you're probably going to be a part of creating that. And, just keeping that attitude of, continuous innovation of learning and that's really what makes the job so fun.

And that, there's not going to be any end to that. It'll just keep changing and there'll be more opportunities to learn something new, which is a lot of fun.

hugo: Yeah, I love it. Keep open and keep experimenting.

elena: Exactly. Yes. And come get some ice cream if you want. A

hugo: that's the final call to action, everyone. Next time you're in New Haven, Connecticut, go to Elena's and get some delicious ice cream and try Elena's favorite, which is Let me get this right. It's vanilla, right? Is it vanilla

elena: It could be vanilla chocolate or swirl, I would say. And any flavor is good as a base.

hugo: but it's with chocolate and sea salt.

elena: Yes. It's our special chocolate dip.

hugo: Amazing.

Really good. Thank you so much for coming to share [01:06:00] like a multifaceted journey from tech to ice cream, to lecturing, to political consulting. I think you've had such an interesting career, Elena, and really appreciate you sharing your wisdom and being so generous with your time. 

elena: thank you so much Hugo for having me. It's always a pleasure. And,I love your podcast, so keep up the great work.

hugo: Thank you so much.