SECOND OPINION WITH DR DEW Episode 014: The AI Hiring Tools No One Audits Published: 2026-01-01 Runtime: 38:12 --- [00:00] Today we're looking at something that affects millions of people but that most people don't know exists: AI hiring tools. The systems that decide whether your resume gets seen by a human, whether your video interview gets passed along, whether you make it past the first cut. [00:22] This is a topic where the gap between the marketing and the reality is significant. And it's a topic where the stakes are high—we're talking about people's livelihoods, their careers, their ability to support themselves and their families. [00:42] So let's apply the method and see what survives scrutiny. [00:50] First, the claim. What's actually being asserted about these tools? [00:58] The marketing from AI hiring vendors typically makes three claims. One: these tools are more efficient than human screening, processing thousands of applications quickly. Two: they're more objective, applying the same criteria consistently to every candidate. Three: they reduce bias, removing human prejudices from the screening process. [01:28] The efficiency claim is the easiest to evaluate. Yes, algorithms can process applications faster than humans. That's straightforwardly true. If you define success as "speed of initial screening," AI wins. [01:45] But efficiency toward what end? If an algorithm quickly filters out qualified candidates, it's efficiently producing bad outcomes. Speed isn't a virtue in itself. [02:00] The objectivity claim is more interesting. Algorithms do apply criteria consistently—that's what they do. But "consistent" isn't the same as "correct" or "fair." An algorithm that consistently penalizes candidates with employment gaps will consistently penalize caregivers, people who took time off for illness, and people who were affected by economic downturns. Consistently wrong is still wrong. [02:32] The bias reduction claim is where things get really complicated. This is the claim vendors lead with, and it's the one with the weakest evidence. [02:45] Here's the theory: humans have unconscious biases. They might favor candidates who remind them of themselves, who went to familiar schools, who have names they can easily pronounce. An algorithm doesn't have those biases. [03:05] The problem is that algorithms learn from historical data. And historical hiring data reflects decades of human decisions—including all those biases we're trying to eliminate. If you train an AI on data from a company that historically hired mostly men for engineering roles, the AI will learn patterns associated with "successful candidates" that correlate with being male. [03:35] This isn't theoretical. Amazon built an AI recruiting tool in the mid-2010s. They had to scrap it because it systematically downgraded resumes that included the word "women's"—as in "women's chess club captain" or "women's college." The algorithm learned this pattern because the historical data showed mostly men being hired into technical roles. [04:05] Now, Amazon caught this problem because they were Amazon—they had the resources to audit their own tool. Most companies using third-party hiring AI don't have that capacity, and the vendors aren't volunteering to open their black boxes. [04:25] Let me trace the incentives here, because this is where the accountability gap becomes clear. [04:35] The vendors have a strong incentive to claim their tools reduce bias. It's a selling point. HR departments are worried about discrimination lawsuits, and a tool that promises to reduce legal liability is attractive. [04:52] But vendors have almost no incentive to prove these claims rigorously. Independent validation is expensive. It might reveal problems. And currently, there's no regulatory requirement to disclose audit results before deployment. [05:10] Employers have an incentive to believe the claims. They've already purchased the tool. They've integrated it into their workflow. Finding out it doesn't work as advertised creates a problem—sunk costs, potential legal liability, operational disruption. It's psychologically easier to assume the tool works than to investigate whether it doesn't. [05:38] Candidates have the strongest incentive to care whether these tools work fairly—it's their jobs on the line. But candidates have almost no visibility into what's happening. When you apply for a job and never hear back, you don't know if a human rejected you or if you were filtered out by an algorithm. You don't know what criteria were used. You can't appeal or ask questions. [06:08] And regulators? They're trying, but they're behind. New York City passed Local Law 144 in 2023, requiring bias audits for AI hiring tools used in NYC. It was the first law of its kind in the US. But the implementation has been rocky—weak audit requirements, limited enforcement resources, and vendors finding creative ways to technically comply without meaningfully improving. [06:40] So we have a situation where the people with the most at stake have the least power, and the people with the power have the least incentive to verify that these systems actually work as claimed. [06:58] Let me zoom in on one specific type of tool that I think deserves extra scrutiny: video interview AI. [07:08] These are systems that analyze recorded video interviews—either live or asynchronous—and claim to assess things like communication skills, personality traits, emotional intelligence, and "culture fit." [07:25] Some of these tools analyze facial expressions, tone of voice, word choice, even body language. They produce scores that feed into hiring decisions. [07:38] The empirical basis for these tools is thin. Really thin. [07:45] The vendors cite internal validation studies, but those studies are rarely peer-reviewed or independently replicated. When independent researchers have looked at similar systems, they've found problems with generalization—models that work in one context don't transfer well to other contexts. [08:08] There are also serious questions about proxy discrimination. If a system analyzes speech patterns, it may effectively discriminate against non-native speakers or people with certain regional accents. If it analyzes facial expressions, it may perform differently for people with different facial structures, skin tones, or those who are neurodivergent. [08:35] The EEOC has started paying attention. There have been settlements. But enforcement is slow, and by the time a pattern of discrimination is proven, thousands of candidates may have been affected. [08:52] Now, I want to be careful here. I'm not saying all AI hiring tools are bad or that they can never work. I'm saying the evidence that they currently work as advertised is weak, and the accountability mechanisms that should exist don't. [09:15] Let me give you the plain-language translation. [09:20] If you're a job seeker: know that these systems exist. When you apply for a job at a large company, there's a good chance an algorithm is making the first cut. This doesn't mean you should try to "game" the system with keyword stuffing—that often backfires. But you should know that an automated rejection doesn't necessarily mean anything about your qualifications. Sometimes you were filtered out for reasons that have nothing to do with whether you could do the job. [09:52] In some jurisdictions, you have the right to ask whether AI was used in a hiring decision. In NYC, employers must notify candidates about automated screening. In Illinois, there's a law about video interview analysis. Know your local rules. [10:12] If you're an employer: push back on your vendors. Ask for audit documentation. Ask what the training data looked like. Ask whether independent researchers have validated the tool. If the vendor can't or won't answer these questions, that tells you something important about how much you should trust their claims. [10:38] Don't assume that buying a tool transfers your legal liability. If the tool discriminates, you're still the one doing the hiring. "The AI did it" is not a legal defense. [10:55] If you're a vendor: this is your opportunity to differentiate. The market is going to require transparency eventually—regulators are moving in that direction. The vendors who proactively provide rigorous audit evidence will have a competitive advantage when that shift happens. [11:18] And if you're a policymaker: Local Law 144 was a start, but the audit requirements need teeth. Independent audits, not vendor-commissioned audits. Standardized reporting so that employers can compare tools. And penalties significant enough to actually change behavior. [11:40] Let me close with a broader point. [11:45] This episode isn't really about AI. It's about what happens when consequential decisions get automated without adequate accountability mechanisms. AI is just the current technology creating this problem. The pattern is older than that. [12:05] Every time we automate decision-making, we face a choice: do we build in the accountability mechanisms from the start, or do we add them later after harm has been documented? [12:20] We keep choosing the second option. And the people who bear the cost of that choice are usually not the people making it. [12:32] What changes after watching this? You'll know what's verified—that AI hiring tools exist and are widely used. You'll know what's inferred—that the bias-reduction claims are plausible but unproven at scale. And you'll know what's performative—the marketing language that sounds rigorous but isn't backed by rigorous evidence. [13:00] That's the Second Opinion for today. [13:05] If this was useful, check the show notes for sources. Everything I cited is linked. If you think I got something wrong, email me—I'll update the page with corrections. [13:20] Thanks for watching. I'll see you in the next one. [13:25] [END] --- Transcript by: DeWayne Lehman Reviewed: 2026-01-01