How do you bring your entire organization along for the AI ride while inspiring confidence and flexibility amid fast evolution? Hannah Calhoon, VP of AI and Head of AI Innovation at Indeed, shares how she’s helping 11,000 employees embrace AI as a partner in their work.
How do you bring your entire organization along for the AI ride while inspiring confidence and flexibility amid fast evolution? Hannah Calhoon, VP of AI and Head of AI Innovation at Indeed, shares how she’s helping 11,000 employees embrace AI as a partner in their work.
Host Francois Ajenstat gets things going through an exploration of how AI is currently transforming Indeed in both its offerings and the way its teams work every day. Hannah explains a focus on balancing rapid experimentation with thoughtful change management, and why building role-specific skills has been key to adoption thus far.
You’ll hear how Indeed trains teams across engineering, sales, and marketing to use AI tools effectively, how a “Lego block” approach helps them stay flexible in a fast-moving space, and why human judgment still sits at the center of AI-powered hiring.
Hannah also shares lessons on creating safe spaces for experimentation, overcoming resistance to new tools, and leading large-scale change without losing trust. Whether you’re scaling AI in a global enterprise or navigating your first AI initiatives, this episode offers a practical blueprint for transformation at speed.
—
Guest Bio
Hannah is a VP of Product at Indeed, where she leads the AI for Indeed team. Her team focuses on enabling everyone at Indeed to effectively build and use AI tools and preparing the organization for the Future of Work. She also chairs the cross-company AI steering committee, which is tasked with ensuring Indeed is moving quickly and responsibly to deploy AI to drive business outcomes.
Previously, Hannah led Indeed's long-term innovation team and built out Indeed’s GenAI Lab. She has focused her career on building innovation systems that deliver benefits to people's daily lives—initially in pharma and medtech and for the last 10 years in software, with a focus on fintech, legal tech, and HR tech.
Hannah holds degrees from Harvard College and Stanford's Graduate School of Business. She serves as a Venture Partner for Purpose Built Ventures and sits on the board of edtech nonprofit Economic Mobility Systems.
—
Guest Quote
"We sort of think of the world like we're building an architecture with Lego blocks. And so you've got your moderation block, and right now we have one or multiple solutions, either first party or third party, for moderation. But we're building the system so that if two years from now there's a much better way to do moderation, we can effectively pull that block out and stick a different block in. So we are architecturally optimizing for flexibility under the assumption that the choice that’s right today is almost certainly not going to be the choice in three years." – Hannah Calhoon
—
Time Stamps
00:00 Episode Start
01:25 Hannah's AI role at Indeed
04:45 How to implement AI tooling internally
08:30 Overcoming resistance and gaining trust
11:40 Think of AI strategy like building Lego
13:40 You don't always need the latest and greatest
16:30 Who should be responsible for AI strategy?
18:50 The unintentional impact of AI
21:50 Supercharging humans with AI tools
26:15 Change management in driving AI adoption
29:30 Is the danger of bad actors worth losing sleep over?
35:10 Hannah's "Oh Sh*t Moment"
—
Links
0:00:00.2 Francois Ajenstat: I just got tingled. I got super excited about this. I want to lean in. Can we call this Vibe hiring or vibe? Let's coin the term on this podcast right now.
0:00:12.9 Hannah Calhoon: Vibe hiring.
0:00:13.7 Francois Ajenstat: Vibe hiring. I love it.
0:00:15.2 Hannah Calhoon: Yeah. I don't know. And my marketing team is like, oh, God. Oh, God.
[music]
0:00:25.5 Francois Ajenstat: This is Next Gen Builders, the show for the growth and product leaders of tomorrow. Everyone's talking about AI transformations, but how do you actually teach 11,000 people not to be afraid of robots? Today, we're going to go behind the scenes at Indeed to see how they get their employees AI ready. Spoiler alert. It's not as simple as just flipping a switch, it takes deploying AI that actually helps people instead of replacing them and combining the best of both humans and machines to move the business forward. Joining us today to talk through it all is Hannah Calhoon, VP of AI at Indeed and head of AI Innovation. Welcome to the podcast, Hannah.
0:01:13.3 Hannah Calhoon: Thanks so much. I'm excited to be here.
0:01:15.5 Francois Ajenstat: Well, really excited. I mean, let's just dive right in. You've got a pretty interesting rollet, Indeed. I don't think I've heard anybody with that title before. Tell us what you're up to.
0:01:25.7 Hannah Calhoon: So listen, Indeed, I think like many companies has been trying to figure out over the last couple of years what AI means for us. And for us, there's sort of three different lenses of impact. There's first, this big question about what does AI mean for the labor force and the job market? Lots of talking about that. Indeed is the world's largest hiring marketplace. We've got more than 600 million job seekers, 24 million jobs, 60 countries. And so we have this very interesting front seat to the global economy and how AI is starting to impact labor and roles and all of these sorts of things. So there's this question for us, there's a macro question, which is, what is AI going to mean for sort of the context in which we do business? The second level, of course, is what does AI mean for our products and services? We help people find jobs and we help employers find talent. And both of those are processes that AI has a tremendous opportunity to improve and disrupt. And so we're thinking a lot about how do we leverage the best pieces of AI to make it easier and faster and more enjoyable to chart a career or to find the right person to fill a role. And then at the third level, the sort of most micro level, there's this question about what does AI mean for us as a company and the way we do our work and for our employees experience of working at Indeed and being part of Indeed. And how do we make sure that we are taking advantage of everything that AI can do for us in terms of business productivity and outcomes and also just making work better for the people who work for us. My job is fun because I get to spend time thinking about all three of those levels every day and also the places that they intersect with each other.
0:03:13.7 Francois Ajenstat: I mean you're really partnering across the company to figure out how Indeed uses AI, but also how does the market use AI too? Or how does that work exactly? How do the two work hand in hand?
0:03:26.1 Hannah Calhoon: Yeah. We are, I think like many companies working to bring AI into our products and services in ways that are thoughtful and actually value additive. So, job seekers and employers don't want a bunch of AI hype, they want AI that's actually going to make the product experiences better for them. And I think we've done a lot in the last year and a half to roll out new features and we're really excited by some of those results. And then also, like many companies, we also recruit talent and manage talent and try to motivate talent. And so it's really interesting to be able to watch, sort of think of it both as a product and also as an experience that we're going to put our people through and to be able to test some of those same product experiences, some of those same patterns for sort of AI human interaction inside our own operations.
0:04:19.4 Francois Ajenstat: And that's great. I mean, basically, your team becomes almost a petri dish for what's going on outside and you get to learn and experience what others are doing too.
0:04:28.9 Hannah Calhoon: Yeah. We're really lucky and it's a lot of fun.
0:04:31.1 Francois Ajenstat: That's really cool. How far along are you in the transformation of Indeed, maybe internally in adopting AI and maybe what are things that have worked and other things that have been maybe more challenging?
0:04:45.0 Hannah Calhoon: Yeah. So, I would say that we're in the middle of our journey. We spent a lot of last year doing smaller scale experiments and pilots, trying to get tools into the hands of all of the folks that work at Indeed and giving them space and guardrails and sandboxes so they could play around and tell us what was useful. And I think at this point, we have a series of hypotheses around the largest, most interesting opportunities, and now we're really trying to go after those at scale. So a lot of our focus last year was, our sort of key KR was usage. Like what percentage of Indeedians are using an AI tool in their day-to-day work on a weekly basis, for example. This year, we've really shifted gears and said, okay, usage is great, but usage doesn't drive business impact. And so we've said, okay, in a core set of verticals where we think there's a lot of opportunity, we not only want to drive usage, but we really want to measure, like what does the usage get us? Are we saving people time? Are we improving the quality of decision-making? Are we improving customer satisfaction scores? For example. Like what are the things... Like AI is great, but why is AI great? But we aren't at a place where we've like rolled that out at scale and it's working and it's perfect and I can like set up shop and go home. And so we're not at the end. We're squarely in the middle.
0:06:07.0 Hannah Calhoon: There's a couple of things we've learned, but one of the biggest ones has really been around what does it take to get the maximum value of your investment in these tools out of the tools and out of your workforce? We have launched a bunch of different AI tools, and every time we release a new tool, we create a chart where we look at what percentage of the usage is driven by what percentage of the relevant employee population? And out of the gate, it is a beautiful Pareto curve where 10% or 20% of the early adopters are driving 80% or 90% of the usage. It's exactly what you would expect. And getting from that Pareto curve to deep integration of a tool into your workflows and therefore maximum impact on your business requires going to all of the other people and actually being able to sit down with them and say, here's why this tool is useful for you in your job, in your workflow.
0:07:07.5 Hannah Calhoon: Here are three or five ways that you should try using it as part of your job. And that requires very function specific, role specific training that takes into account the work processes and the journeys that our employees go through every day. And so we have done, we've got a great sort of like AI 101 foundational curriculum that's designed for everybody at Indeed. And then we've got a whole host of function specific hands on training programs where we go and we say, okay, we're going to spend a lot of time just with our software engineers. We're currently rolling out this AI coding essentials program. And over the course of three months, we're going to put 2,200 of our SWEs through hands on training to help them make better use of the various engineering AI tools that we've rolled out across the company. And then we're going to go do a version of that in sales, and we're going to go do a version of that in finance, and we're going to go do a version of that in marketing. Because even a horizontal tool that's sort of generically useful, think about a Gemini or a ChatGPT or a [0:08:15.8] ____ , one of these things, it's useful for different people in different ways. And if we put the burden of figuring that out on our employees, it's just going to be a really slow process and we're trying to move fast.
0:08:27.8 Francois Ajenstat: I agree. And where have you seen the resistance? Is it resistance? Are people overwhelmed? Are there just too many tools and it's changing too much? Why aren't they all leaning in and taking advantage of all this stuff?
0:08:43.3 Hannah Calhoon: Yeah. I think some of it is there are a lot of tools. And one of the blessings and the curses of working in this space right now is that there is a new cool thing every other week. And so folks do need support navigating the options. And we have tried as a company to take an approach that says, we're not going to say you can't use something, but we are going to pick some things that we think are the highest quality enterprise bets and we're going to provide support and training and paved paths around using those options. Because we've got our explorers and our explorers want to go and touch everything and try all the stuff. And then we've got the people who just want to do their job and they're like, I just need you to tell me what to use to solve my problem. I want to learn how to use it and I don't want to worry about it again. And so I think a lot of it is actually is that sort of knowledge and accessibility. There's a willingness to try a new tool that's going to make their job better, but they want you to be more directive about what the tool is and then also it has to make their job better.
0:09:52.0 Hannah Calhoon: And certainly, if I go back a year when some of these tools were a lot newer, they were buggy and they hallucinated and people sort of said, gosh, is this actually less work? And so part of our strategy which has been a lot of small controlled pilots to understand the edges of what the tools are capable of and where the highest ROI is, has also been validating like, is this a thing that's solid enough that rolling it out to the broader employee base is actually likely to be successful? And contrary maybe to what sometimes people think, we have tried things and passed. We've gone through the early pilot with the vendor, we've looked at the data, and we've said, gosh, we love the vision, we love the roadmap, we think this functionality is going to be really valuable for Indeed when it's ready. But right now, it's not ready, and it's not worth the sort of change management risk to roll out a subpar product to our employee base because it will erode some of that trust and some of that willingness to experiment, and we want to hold on to those arrows until we can use them in the places that we think will be impactful.
0:11:07.6 Francois Ajenstat: I mean, that makes sense. On that note, the AI market is evolving at a speed I've never seen before. There's always new tools, the models are changing, there's new capabilities, there's new acronyms like MCP and A2A, whatever. It keeps changing so fast. For you, as you're building the roadmap for Indeed, how do you plan on a moving target where everything constantly is evolving? Can you plan? Or how do you approach it?
0:11:41.4 Hannah Calhoon: Yeah. It is both really fun and really hard. One thing that we have really leaned in on, and I have to credit a colleague of mine who leads our engineering and science for AI platform. His name's Chris Johnson because he coined the term the Lego block architecture. But effectively, the strategy we have taken is to say, okay, we have a set of capabilities that we know we're going to need architecturally. We're going to need foundational models. We're going to need moderation. We're going to need evaluation and real-time auditing. We're going to need memory. We're going to need systems for interoperating between agents eventually. So many of the platforms that we build on and the platforms we're building involve agentic capabilities. We know eventually agents are going to have to talk to agents, and that handoff and that communication is going to have to have a certain set of components around identity and security and information transfer. So we know the pieces of things we're going to need, and there are a growing number of options to fill any of those pieces. And so we sort of think of the world as like we're building an architecture with Lego blocks, and so you've got your moderation block. And right now we have one or multiple solutions, either first party or third party, for moderation, but we're building the system so that if two years from now there's a much better way to do moderation, we can effectively pull that block out and stick a different block in. So, optimizing for modularity, optimizing for flexibility, under the assumption that we do want to be able to make investments, maybe not 10-year investments, but multi-year investments in platform capabilities because we need to move fast, we need to move to scale, but also we may not make the choice that's the right choice today is almost certainly not going to be the choice in three years.
0:13:32.8 Francois Ajenstat: That's for sure.
0:13:33.8 Hannah Calhoon: And we all just need to be comfortable with that. We are architecturally optimizing for long-term flexibility. I think the tension though, is that because things are changing so fast, there's also this deep temptation to hop to the next thing really quickly. And part of what we're starting to see is, like AI coding assistance is a great example. There are so many of them now, and every week someone comes out with a new one and says that it beats benchmarks in a certain set of ways. My personal hypothesis is that most of these platforms, they're eventually going to get to parity. And so somebody being a little bit ahead now or a little bit ahead in a month is maybe from a long-term perspective, not that important. And so picking a couple of partners who we think have the right philosophy, have good design principles, are really great to work with and saying, okay, your services may not always be at the very top of the benchmark, but we have a lot of confidence that over time you'll be at parity or better than any place else we could go. And so we're not going to do the thing where we hop from assistant to assistant and try to optimize every way is actually going to simplify our lives and help us to drive more impact over time. Because every time we bring out a new tool, new onboarding, new integration, new training, new thinking about how it connects to data sets, new connection to the MCP server. And so there is, it's like you both need to be super flexible and you need to be willing to make some bets and hold your ground for at least some period of time so that you're not just being sort of buffeted around by the winds of change.
0:15:19.0 Francois Ajenstat: I mean, that's actually really great advice. I've also seen the counter argument where some people feel paralyzed by the pace of change because they don't know if what they're going to pick is the right one, so they just do nothing. And I actually think that may be creating more pain for the long term.
0:15:38.7 Hannah Calhoon: 100%. I have told my team multiple times at this point, I need you to make a choice. We're going to make a choice. If it's not the best choice, that's on me. But we're going to make more impact on the business if we choose a good or even a great but not perfect solution now and we start driving towards execution and scale of implementation than if we wait until we're absolutely sure which of the 12 alternatives you're considering is the very, very best one. And so we sort of actively have that conversation all the time.
0:16:14.6 Francois Ajenstat: How much are the tool selections or the tool excitement coming bottoms up organically from the team members versus you as a central body or central team starting to set the guardrails and guidance for the groups?
0:16:30.3 Hannah Calhoon: Yeah. Definitely some of both. I would say every week we have a bunch of Slack channels that are devoted to AI exploration and innovation, and every week there's somebody in the AI channels with, oh my gosh, I tried out this really amazing thing and can we get it at Indeed? Which is great. You have this amazing source of ideas. I would say our team does a lot more of thinking about like, okay, somebody's come with this tool that they're really excited about. Like, let's understand what's the use case that the tool is actually solving? Like what's the problem? And do we have something in the existing portfolio of solutions that solves that problem well? And if so, this is an interesting data point around training and enablement and helping people understand how to get the best use out of our platforms. But if this is sort of net new, or if it's something that we can do somewhere else in the platform, but this is meaningfully exponentially better, then there's a really interesting question around, okay, how do we understand very quickly what the business case would be for bringing on a new tool? And how big a problem is it?
0:17:34.3 Hannah Calhoon: Is it a problem that drives a measurable business outcome? How many people does it apply to? What's the cost of the thing? All of that type of stuff. So it's definitely both directions, and to be transparent, it's also the total opposite way, which is aggressively top down. Somebody at the board level or in the C-suite was chit chatting with another friend of theirs who runs a company and they said, oh my gosh, we love this tool. And then you get the email from the CEO that's like, hey, when are we rolling this thing out? And you're like, oh, okay, that's interesting. I think part of what's really fun about the space is that everybody is exposed to this stuff and thinking about this stuff. And so, the ideas can come from anywhere.
0:18:18.6 Francois Ajenstat: By the way, that happens to me all the time. I get so many pings about new tools. It's fun, it's inspiring, but it also can be overwhelming. Have you found that as people are using these tools, it's inspiring them in different ways of building capabilities for Indeed's user base? If you think about the features they build, is there a correlation between what they use to build versus what they build?
0:18:49.7 Hannah Calhoon: Yeah. That's an interesting question. I mean, I do think so. So, we started our exploration of what to build for job seekers and employers probably a good year before we really started seriously thinking about what we wanted to build or buy or otherwise partner to develop internally. And I don't know if that was the right order or not. And so I think for a while, the learning was moving in the other direction. We were like, oh, we've discovered that this sort of problem is really hard to solve for job seekers, which means it'll probably be really hard to solve for our employees too. Or, oh, we know that job seekers as they interact with our chat agents really like these sorts of formats. And what can we learn there from a UX perspective? I think part of what we're seeing now as we roll things out internally is it is helping people sort of, as people are having the personal experience of how transformative it can be for workflows, it does sort of set off that spark of, okay, just because there's always been this set of steps in the process of going through an interview, we don't have to adhere to that sort of very linear process necessarily anymore.
0:20:07.1 Hannah Calhoon: They're like, oh, if I can have this experience where I, as the product manager, am able to pull up and create easy visual content prototypes and then use those prototypes and sit in Cursor with my engineer and we can all, like me and my UX lead and my tech lead, can sit together and do live prototyping in Cursor where we're building something that we can all talk about and react to, maybe that's an interesting mental model for how recruiters could work with hiring managers to think about a pool of talent for a role. Maybe it doesn't have to be the recruiter goes out and writes a job description and interviews 300 candidates and screens them down and does all of this work and then comes to the end of that process and hands the hiring manager a folio with five resumes in it. Maybe that implied linearity and the real division between those roles actually isn't what we're going to see in the future. And maybe it still is. We don't know. But I do think it creates some really interesting thought experiments around the ways in which the world might change and that it's always easier to do those experiments when you've had that experience yourself of feeling the excitement and feeling the viability and realizing that maybe there's different things that could be better.
0:21:24.4 Francois Ajenstat: I just got tingles. I got super excited about this. I want to lean in. Can we call this vibe hiring or vibe? Let's coin the term on this podcast right now.
0:21:37.3 Hannah Calhoon: Vibe hiring.
0:21:38.2 Francois Ajenstat: Vibe hiring, I love it.
0:21:39.7 Hannah Calhoon: Yeah. I don't know. And my marketing team is probably having a... They're like, oh God, oh God. Yeah. I think it'll be interesting to see. I mean, one of the things that we see across both our external products and our internal tools that's so interesting is the value, like the value of deep, rich context that lives in people's brains. And I think sometimes when you talk to people or you hear people get up and talk about how AI is going to transform work or jobs, they talk a lot about the sort of specific tasks or analytic pieces that AI is great at. And then they look at job descriptions and they say, well, this job is mostly doing a bunch of these things. And so AI is going to come in and be really disruptive for this role. And it's not that AI can't be helpful for parts of the job, but so many jobs have this sort of deep contextual component. And to pull an example from our product experience, there are probably more, but there are sort of three basic ways if you are looking for a job that you as a job seeker might be invited to apply for a job on Indeed. And so one is, we might use AI to identify jobs that could be a good fit for you and send you an email and say, Francois, we think you should apply for this job. And you get this email and you click on the link and you apply some of the time and you're a decently good candidate. And so like that works and it's super cheap and it's very automated and it doesn't involve humans. And it's very, very efficient, if we were optimizing around efficiency. There's a second version of how you might get invited, which is that a human recruiter sits down at Indeed and they search through the millions of job seeker profiles on our database and they find people they think are interesting and they send them emails to invite them to apply for the job. And that is more effective, slightly more effective than the AI version, but it is very, very time and resource intense. But you can understand why it would be better. The recruiter has a really good sense of who they want and so they're being more picky and all of those sorts of things.
0:23:47.1 Hannah Calhoon: And when you get an email directly from another human saying, hey, apply to this job, that's more compelling than getting the email from a computer. We now have a product that blows both of those other things out of the water, actual literal multiples of improvement in terms of the likelihood that a high quality candidate who will eventually be hired applies. And that's that the AI does all of that work to pre-screen and identify possible candidates, and then they provide them to the recruiter with little AI generated blurbs about why the candidate could be interesting. The recruiter goes in and clicks on the ones that they're interested in and sends the emails. And those people, the folks who have been invited by the AI and human working together are the very best. That's the best of the patterns. And I think it's a really lovely example of the fact that the AI can go out and do, it can scan 600 million profiles in the job database. It can do the mashing and the comparison. It can think about what we know about the job seekers wants and needs and their profile and their experiences.
0:24:49.9 Hannah Calhoon: But at the end of the day, there is so much more in the recruiter's brain about what a great candidate for the role is going to look like than they have put in the job description. That having that person who looks at everything and applies that additional context and that additional human judgment is still something that we have no idea how to mimic technologically. And I think increasingly we're going to find that many roles, even roles where we don't necessarily think about the amount of human context and judgment and connection that are necessary, still have these very, very core components where the AI can be super helpful, but only to a point. I think we're seeing that in sort of both sides, both the internal and the external, and we're trying to figure out how do we pull all those insights together on both sides and make good choices for all of our stakeholders.
0:25:40.1 Francois Ajenstat: I mean, that's super compelling. I mean, it is another example of a human in the loop drives better outcomes. AI can remove some of the drudgery that's out there, but it's a human and AI connecting together, which I think is really important. So we've talked a lot about technology and as technologists, we love technology. But as leaders, building culture is extremely important and it's another beast entirely when we talk about AI. How do you think about leadership, change management, getting people on board with this massive transformation we're all getting into?
0:26:14.8 Hannah Calhoon: I was literally talking to a set of people last week and I was giving a talk and the headline message of the talk was, technology is less than half the battle. Because just having cool tech tools, like I love a cool tech tool, but that is not actually the thing that is going to drive meaningful business impact at scale. I think there's a couple of things that we have found have worked for us. One is, you need the leadership buy-in. You need leadership to say, this is an important trend, it's coming, it's coming quickly. We need to all collectively be focused on taking it forward. And we have as a company every fiscal year, a set of four or six top priorities. AI in its external and internal implementation has been on that list for the last couple years. This year it's like two of the points on the list. And so we've said, okay, we as a company are committed to making this transition. But also, as you and I have talked about, execs being really excited about something is sort of necessary but not sufficient. You also have to get everybody in the employee base excited about it.
0:27:29.2 Hannah Calhoon: And I think some of the things that we have tried there that have been somewhat effective are, we are creating, trying to create spaces, safe spaces for experimentation, communities for folks who want to push the envelope and are trying stuff to connect with each other and to share best practices. We have these spotlight series where we go find people who have implemented AI in interesting ways in parts of the business. And we do little blasts on them in newsletters and in Slack and in other ways. I think we've tried to be straight with people and transparent about the ways in which we imagine AI might change jobs and work. And we've leaned into the fact that some people are nervous, and that there are concerns about what does this mean for me or my role? And we've really said, listen, there's a lot that's coming that we don't necessarily understand, but we believe really strongly that every single one of you, if you're a software engineer or a customer support rep or you're in legal or whatever it happens to be, every one of you is going to need AI skills and AI comfort and familiarity and the ability to use AI tools successfully for your career to continue from here.
0:28:38.8 Hannah Calhoon: The world is changing. You're going to need to be ready. We're going to make you ready. We're going to put every single one of you through function-specific, role-specific training. We're going to give you the best and breed tools that we can find. We are going to make sure that you are making this transition into this new AI-powered economy with us. And we don't know what that's going to mean over time for how many people will work at Indeed five years from now. Can't control the future. What we can control is trying to implement technology in a way that is worker-centric, that is empowering. And as we make choices about which tools to roll out or any of those sorts of things, make sure that we're actually really listening to your feedback and that we're only rolling things out when we think they're actually going to be useful and helpful and accessible and all those things, So that we're not going to have to convince you. Once we show you how it works, we won't have to convince you to use it. You'll want to use it because it's helpful and it reduces that toil and that drudgery and all that stuff we talked about earlier.
0:29:34.6 Francois Ajenstat: That makes sense. So, with great power comes great responsibility.
0:29:41.6 Hannah Calhoon: Spider-Man.
[laughter]
0:29:44.1 Francois Ajenstat: Had to bring that reference in. But with all great technology advances, there's amazing things you can do, but there's also the negative side where some people are abusing it or there's other concerns that come up. Have you seen situations ,maybe externally or internally, whatever that is, where people are playing the game and using their own AI against your AI or finding different ways of bypassing the AI you've built?
0:30:14.7 Hannah Calhoon: Yeah. I think there are always people trying to game a system. And AI is just throwing more fuel on the fire around trust and safety and moderation and verification and all those sorts of pieces. Folks are like, well, are people using AI to beat resume screeners? And it's like, well, there used to be pre-generative AI, this practice that I didn't really discover until I got to Indeed, which tells you how up on things I am, where people, ATS systems historically have used keywords for more traditional machine learning matching to look for candidates. And so people would do this thing where they'd write their resume and then in white text at the bottom so you couldn't see it, they just fill in dozens and dozens of words that they thought might be useful for ATS matching.
0:31:00.5 Francois Ajenstat: Really?
0:31:01.2 Hannah Calhoon: Yes.
0:31:01.3 Francois Ajenstat: Oh my God! That's amazing.
0:31:03.9 Hannah Calhoon: So this has always been a thing, I guess is maybe the point. And yes, now you've got these situations where employers come to us and they say, well, doesn't every job seeker with a pulse have a great, beautiful ChatGPT resume? Or I've been interviewing folks for software engineering jobs, and I think that they are using AI to beat their coding assessment tasks. And so it does create a new level of difficulty around going out and being able to really verify information. It's going to create some new patterns we're going to need to see in terms of how do we assess talent? How do we collect information? How do we know people are who they are? We don't want people to suddenly flood employers with 10 million AI-generated job applications. That's not going to be good for anybody.
0:31:54.7 Hannah Calhoon: But I do think at the end of the day, we believe that we're going to be able to provide the best experience for job seekers and employers if we have that deep, rich context on what both of them are looking for, and we can use that context to suggest matches.At the core of what Indeed does, we're a matching marketplace. And one of the things I think is so cool about generative AI is that it gives us sort of new modalities to collect more of that context and then utilize it effectively. So if I'm a job seeker, let's say I've been working a bunch of retail jobs. I've been a barista. I've worked at the Target. I've been doing all of these sorts of things while I have in parallel been doing night school to get a degree in content marketing. In the old world, I put my resume onto Indeed, and it reads all the stuff I've been doing, and it just naturally is going to suggest a bunch of jobs in retail and service. It's looking at my experience, and it's matching me with jobs. But I'm trying to make a shift into content marketing. And I want it to underweight all of those things and overweight the internship experience of a portfolio project that I've just done. And in a world of generative AI, I can tell it that, and it can understand that and remember it, and it can rethink effectively the relative weight it's giving to different parts of my profile.
0:33:20.3 Hannah Calhoon: Or like humans change their minds all the time. Maybe I have been searching for content marketing jobs in New York, and I've suddenly found out that I have a family member who is going to need support. And so, even though for the last three weeks I've been doing nothing but searching for content marketing jobs in New York, I like, as of tomorrow, what I'm looking for is a job that pays a minimum of X number of dollars in Madison, Wisconsin. Traditional ML models are looking at my past history of behavior, and they're like, you want a content marketing job in New York? And so they're going to keep trying to serve me content marketing jobs in New York, even though my life and my needs have changed. And so I think there's this really wonderful opportunity with generative AI for us to have such a deeper understanding of skills and experience and motivations and which of those things matter to individual job seekers at any point in their life, which then allows us to serve them increasingly relevant, increasingly useful jobs. And so maybe the universe gets cluttered with a bunch of nonsense job seekers and nonsense jobs and nonsense applies. And so you've got the problem of trying to find the needle in the haystack. But if you have really good data about the needle, finding the needle and finding the job that is the perfect job for the needle is actually very doable. And so we are doing a ton of work to invest in trust and safety and identity verification and all of the things to keep abuse and fraud out of the platform. And also we think that this ability to really delve more deeply into context and needs on both sides of the marketplace will help us fight the sort of AI clutter in a different and in some ways more productive way.
0:35:02.2 Francois Ajenstat: I'm just inspired. I love the possibilities and I love the approaches that you guys are taking and just really thoughtful. Okay. Hannah, I have one last question. I ask all my guests this question. It's one of my favorites, which is really your oh shit moments. When you think about your career, all the things you've done, there's always highs and there's lows and there's incredible learning moments. So, what is your oh shit moment?
0:35:33.4 Hannah Calhoon: Yes, there have probably been more of them than I am proud of. But the one that comes to mind... So, a couple of jobs ago, I started a venture studio in New York. And for those who aren't familiar with the venture studio model, you basically try to create a place where you go out and find founders and help them start companies. And we were specifically focused on helping people start companies where their target consumer was low-income households, because we thought that was sort of an interesting gap in the existing capital market. And it's really hard for a bunch of reasons that are not worth going into now. But we had a founder who came to join the first cohort of our studio, and he was spending a long time at the SNAP office. SNAP is the Supplemental Nutrition Assistance Program, better known as food stamps. And he would go every day and talk to people for sort of UXR purposes. And he was noticing that there were hundreds of people standing in line waiting to turn in their paper forms while they were playing Candy Crush on their smartphones. And he's an ex-Facebook, LinkedIn guy.
0:36:40.6 Hannah Calhoon: And he was like, well, it seems like there must be a better way. And for what it's worth, the company is actually, it's called Propel. They now do other things related to SNAP, but they help about 5 million families across the country work towards financial stability, and they're cool, and you should check them out. So they were like, okay, we're going to create a way for you to submit a SNAP application on our phone in New York. And they started doing that, and they'd been doing it for a couple of weeks, and we got a cease and desist letter from the city of New York, which was definitely an oh shit moment. Having the government send you a legal document is never super awesome. We were probably six months into running the studio at this point. I still wasn't entirely sure I knew what I was doing, and I was like, oh, oh, okay, we have made some mistakes. So we had made two mistakes, and I think they're probably both worth mentioning. The first was, legitimately, there was a problem with the applications, which was that New York was a bit behind the times and had not yet approved electronic signatures for benefits documentation.
0:37:44.9 Hannah Calhoon: And so the apps that were being submitted to the city were invalid because they didn't have physical signatures attached. And, listen, to their credit, the team went out and found every person who'd submitted an application and drove to their house and had them sign a piece of paper and got this sorted. But it was a really good reminder of the fact that to the extent that tech talks a lot about moving fast and breaking things, if what you're moving fast on is people's government benefits that are going to allow them to buy groceries for the month for their kids, you can't fuck that up. So you got to figure out how to move fast and also protect the customer base. Part one, we screwed up. Legitimately, they couldn't take the signatures. We had to get that fixed. The second thing was just contrary to my maybe super overly optimistic view, the government was not super excited about some random crew of people in Brooklyn bringing technology to solve a problem that they owned. And they didn't control the app. They didn't love the optics of us writing pitch decks where we shone a light on how broken the system was.
0:38:53.8 Hannah Calhoon: Nothing about this was exciting to the city of New York. And it was a good moment as well because, again, in sort of traditional tech VC land, if you're the upstart and you're disrupting, maybe you can disrupt the incumbent out of business. And so like, of course, they're not happy, but who cares. But in the government, like the city of New York wasn't going anywhere. Disrupting the city of New York and having them be pissed at us for the rest of this company's history was a terrible idea. And I think about that a lot as I'm thinking about internal organizational change, because what we're trying to do at Indeed is make this really big, really fast technology shift. And Indeed is still going to be there when I'm done, when I'm long gone. And so this notion that you're just going to push through with disruption and the righteousness of being correct about the way the world is going it's not a helpful attitude. You have to make other people excited about the change. You have to get other teams at Indeed to want to own the change and take on the change.
0:39:55.8 Hannah Calhoon: And when you go to them and say, you need to change your process, you need to change your tech stack. They need to believe that that's something that they want and you need to figure out how to do that while making them look good. And so one of my team's core values and the work that we do is outcomes over ownership. And it's so important because our goal is to get Indeed over this mountain. And honestly, I think the way we get there is actually by making a bunch of other teams into absolute rock stars. And if you can like internalize, that you don't need the ownership and the ego, I actually think that really sets you up to succeed in a really different way. And so this was rough. It did not sink the studio. It obviously did not sink us or the company, but it was a really lovely learning moment that I have found I reflect on a lot even though I'm not doing something really different.
0:40:49.4 Francois Ajenstat: That's amazing. And this is why you are where you are now. You're inspirational, you learn, you're a Next Gen builder. I love it.
0:40:59.3 Hannah Calhoon: Thank you.
0:40:59.6 Francois Ajenstat: And thank you for all of your perspectives and ideas and views. We've gone from Lego blocks to Spider-Man and change all the way across. So, Hannah, thank you so much for joining us.
0:41:13.6 Hannah Calhoon: Thank you.
0:41:14.0 Francois Ajenstat: And thank you all for listening to Next Gen Builders. Look out for our next episode wherever you get your podcasts. And please don't forget to subscribe.
[music]