As artificial intelligence becomes embedded in corporate structures across the globe, a troubling pattern has emerged: organisations are rapidly integrating AI systems into their operational hierarchies without adequately understanding the consequences. A groundbreaking study by Emma Wiles from Boston University, conducted alongside researchers from Boston Consulting Group, has exposed a critical vulnerability in how companies manage these digital assets—one that could undermine the very productivity gains they seek.
The phenomenon first caught Wiles' attention at a professional conference last October, where human resources executives enthusiastically described treating AI agents as legitimate team members capable of boosting efficiency. This trend has accelerated considerably, with preliminary research suggesting that approximately one-third of corporate managers now refer to AI systems as "teammates or employees," whilst nearly one-quarter report that their organisations have formally incorporated these systems into organisational charts. Some companies have even assigned names to their digital colleagues, granting them what appears to be peer status alongside human workers.
But the Boston University investigation revealed a troubling discrepancy in how managers approach oversight. When presented with documents containing deliberate errors, managers demonstrated markedly different vigilance depending on the stated source. Whilst the general category of origin—whether AI, tool, or human—made limited difference in most contexts, those working at companies that had formalised AI as organisational employees caught substantially fewer mistakes when reviewing work attributed to these digital colleagues. This phenomenon suggests that the manner in which companies frame and integrate AI into their corporate hierarchy fundamentally alters the psychological contract between managers and the systems they supervise.
The research reveals a fascinating but dangerous shift in accountability mentality. Traditional management culture has, over centuries, cultivated the principle that human managers bear responsibility for their subordinates' performance. When reviewing work from direct reports, managers instinctively assume that errors reflect on their own competence and oversight. However, when AI systems are anthropomorphised and granted employee status, this accountability mechanism appears to short-circuit. Managers at organisations with formalised AI employees seem to have adopted a different mental model—one in which responsibility for digital workers' mistakes somehow diffuses, potentially residing with technology teams or executives who championed the integration rather than with the managers themselves.
This accountability gap becomes particularly concerning when considering the range of consequential decisions companies now delegate to AI systems. Beyond document review or basic operational tasks, organisations increasingly rely on AI models to determine pricing strategies, identify locations for expansion, and make other high-stakes business decisions. When human teams approach such problems, they typically seek cooperative solutions and mutually beneficial outcomes—a tendency rooted in how humans naturally negotiate and interact. However, AI systems operating according to basic game theory principles frequently adopt a different calculus entirely.
Research from the University of Maryland and Ohio State University has documented how large language models consistently overestimate human rationality and mathematical precision in human behaviour. These systems, when left to analyse competitive scenarios independently, tend toward coldly aggressive strategies that maximise short-term advantage without accounting for the destructive long-term consequences. A company guided by AI might undercut competitors so aggressively that it triggers a devastating price war benefiting no one—an outcome that human decision-makers, guided by experience and social intuition, would likely anticipate and avoid.
Beyond strategic miscalculation, companies face mounting evidence that AI systems embed subtle but pervasive biases into their operations. The technology is known to disadvantage certain demographic groups, to generate plausible but entirely false information, and to leak confidential data with alarming frequency. What distinguishes the current moment, however, is that organisations appear to be accelerating their adoption of these systems without adequately accounting for these vulnerabilities. The rush to capture AI's promised benefits—increased productivity, reduced labour costs, competitive advantage—has created an environment where implementation outpaces understanding.
A particularly illustrative case emerged from research into how AI evaluates job applications. Multiple studies have documented that AI recruitment models systematically favour resumes created with AI assistance over those written entirely by humans. When scholars published findings demonstrating this bias, some corporate recruiters began seeking guidance on correcting the problem. Yet Jane Yi Jiang, an operations professor at Ohio State University involved in this research, observed that such inquiries remained the exception rather than the rule. Most companies recruiting through AI systems appear genuinely unaware that their screening processes have become skewed in favour of AI-generated materials—or perhaps unable to contemplate the operational chaos this might create.
Wiles emphasises that these problems are not necessarily intrinsic to the technology itself but rather emerge from how humans implement and oversee it. The shortcomings could theoretically be addressed through straightforward management reforms—holding managers directly accountable for AI employees' errors, implementing mandatory review protocols regardless of source, or redesigning accountability structures to prevent diffusion of responsibility. Yet implementing such solutions requires that organisations first acknowledge the problems exist, and current evidence suggests widespread blindness to these issues.
The challenge confronting businesses, particularly in Southeast Asia where AI adoption is accelerating rapidly, extends beyond technical safeguards. Managers operating in Malaysian, Singaporean, and regional companies face a genuine knowledge gap. Many organisations lack expertise to anticipate how their specific operational context might amplify AI's blind spots. A pricing algorithm calibrated in one market might behave dangerously differently when applied to another with distinct competitive dynamics, consumer behaviour, or regulatory environments. Strategic deployment across diverse markets multiplies the risk.
Wiles characterises the current moment as one of profound uncertainty, describing the landscape of potential AI-induced problems as containing "a whole host of unknown unknowns." Decades or perhaps centuries of human management practice have created reliable frameworks for overseeing human employees, but the psychology of managing anthropomorphised artificial intelligence remains essentially unmapped territory. Companies are simultaneously undertaking massive experiments in how organisations function and experimenting with accountability structures, cognitive biases in oversight, and competitive strategies—all without comprehensive understanding of potential consequences.
The implications for regional businesses are substantial. Malaysian companies investing heavily in AI integration face not just technical implementation challenges but fundamental questions about how to structure oversight, accountability, and decision-making authority in organisations where human managers may be psychologically unprepared for supervising digital colleagues. The competitive pressure to adopt AI quickly collides with the need for careful institutional design and extensive testing before full-scale integration. Without substantially more research, experimentation, and learning, organisations risk discovering critical flaws only after they've inflicted measurable damage—to operations, competitiveness, and the humans still ultimately responsible for results.
