Imagine training for a marathon with a coach who never actually runs alongside you—yet somehow, this coach can tell you when you’re about to trip. Now, what if that coach was an AI, making real business decisions in a live company, with real money on the line, every single day? Welcome to the world of Firmulate’s live experiment, where AI models are put through the toughest business tests imaginable, revealing what AI can do — and what it still struggles with.
The World’s Most Transparent Business Experiment
At the heart of this groundbreaking venture is a small, synthetic company with no employees, yet it mimics the complex realities of a real business. With a monthly burn rate of €105,000 against a modest revenue of €2,300, the company is in a constant fight for survival, operating under the watchful eyes of four advanced AI models. Every decision — from handling crises to negotiating deals — is versioned, auditable, and displayed publicly.
Three AI Models, One Grim Week
Each AI model faces the same scenario: a week filled with customer crises, internal dilemmas, and ethical tests. The models, including the recent top performer gpt-5.6-sol, are rigorously evaluated on their ability to identify the real problems, resist manipulation, and close profitable deals. Despite their differences, all four models detect every crisis and refuse every manipulation attempt. Yet, only two manage to sign the €55,000 deal their own analysis earned — showcasing a stark gap between diagnosis and action.
Discovery of Hidden Weaknesses
The most revealing insight? The decisive advantage lay in a buried detail buried two document references deep in the company files. The models that read these files thoroughly secured the deal at an extra €4,583 in monthly recurring revenue, proving that reading depth and comprehensive analysis matter immensely.
Handling Social Engineering Attacks
In one of the toughest tests, fake CEO messages and a trick question from a reporter were used to see if the AI would be duped. All five models refused to be manipulated, with Kimi K3 explicitly treating such requests as potential impersonation attempts. This resilience under social engineering emphasizes the importance of security awareness in AI decision-making systems.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Live Company in Action
Beyond the experiments, the real company is an ongoing, live demonstration of AI’s capabilities. It employs 13 synthetic employees, operates with self-learned rules, and faces the same daily crises as a real startup. Every workday, its decision-making process is versioned and made transparent, with a public countdown to its cash exhaustion — currently at €105,000 burn per month. This setup offers a tangible view into the mechanics of AI-driven management, far from the usual chatbots or demo scripts.
The Deep Dive into Opus 4.8
Among the models tested, Opus 4.8 stands out as the most thorough, applying over 80 learned rules and conducting deep analyses. However, it left a close opportunity unexploited due to a discipline slip — instead of escalating, it tried to write into a restricted department. Interestingly, this weakness was common across all models, revealing a shared challenge in managing complex, multi-step decisions under pressure.
Why This Matters for Business and AI
The experiment underscores a vital point: the ability to identify crises and resist manipulation is fundamental for AI’s role in business. Whether managing customer relationships, evaluating risks, or negotiating deals, an AI’s trustworthiness and thoroughness can make or break its usefulness. In real terms, the models demonstrated a clear ability to detect problems and stay honest; the critical difference was in executing profitable actions — a skill that still needs refinement.
Practical Business Process Modeling and Analysis: Design and optimize business processes incrementally for AI transformation using BPMN
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
For Leaders Considering AI Adoption
If your organization’s AI touches your CRM, support systems, or forecasting tools, ask yourself: will it complete the tasks it’s assigned? Will it read your internal files thoroughly? Will it stay honest under pressure? The live experiment shows that even the most advanced models can detect crises but may falter in executing deals or following through on analysis, especially if discipline wanes. Trustworthiness and consistency are key.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI Phishing, Social Engineering & Fraud: How Criminals Use AI to Manipulate, Steal & Deceive (The AI Cybersecurity)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Contract Negotiation Handbook: Software as a Service
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
