A CIO’s Guide to Using GenAI in Company: From Pilot to Production
By Nitin Kumar, Director Data Science (GenAI), Marriott International
Every CIO has seen it: a GenAI pilot that works great in a conference room but fails in the real world. When faced with messy business data, strict compliance rules, old systems, and teams that aren’t ready to change, what looks good in a demo often doesn’t work. Many people now call this gap between controlled experiments and real life “pilot purgatory.” It’s a cycle where proof-of-concepts look good but never turn into long-lasting systems. Fewer than 10% of GenAI pilots make it to production, and even fewer show a good return on investment in the first year. The model is not usually the problem. Instead, the blockers are usually about governance, economics, data readiness, and making sure that everyone in the organization is on the same page. A lot of businesses still see GenAI as a technical test instead of the change it really is in how they work and how they think.
In the end, scaling GenAI is not a technical problem; it’s a leadership problem.
The first step to getting out of pilot purgatory is to change how GenAI projects begin. A lot of pilots start by asking, “What can GPT-4 do for us?” instead of “What business problem are we trying to solve?” Beginning with technology results in impressive demonstrations but inadequate business justifications. GenAI works when it starts with a clear business goal, like making more money, lowering costs, making the customer experience better, or lowering risk. Every project needs a business owner who is responsible for it, clear KPIs, baseline performance, and a clear understanding of what is an acceptable mistake. It is much clearer to say, “Cut average handle time by 20% while keeping or raising CSAT,” than to ask for a chatbot. It makes more sense to say, “Answer complicated policy questions in less than 30 seconds with 95% accuracy,” than to suggest a RAG system in a vague way. The project isn’t ready if the executives can’t explain the business value on the first day. Before giving the go-ahead to any GenAI investment, CIOs need to know who will own the results, what needs to be improved, how mistakes will be fixed, and what will happen if the system fails. Clear expectations speed up progress, while unclear expectations lead to costly prototypes.
Governance becomes the foundation, not an afterthought, once the business problem is clear. You can’t add governance to a GenAI system after it’s been deployed; it has to be built in from the start. Enterprise systems need controls that make sure answers are correct, keep harmful or inappropriate content out, keep sensitive information safe, and find and remove PII. These safety measures keep customers and the brand safe, but they also add time and money. A CIO’s job is to find the right balance between risk, cost, and how quickly things happen. High-risk use cases, like giving financial advice, deciding who to see first in a medical emergency, or interpreting a policy, need stronger guardrails, even if it means longer response times. Tasks that are low-risk, like summarizing or writing content, can be done with less strict controls. The right question to ask in architecture is not “Which model should we choose?” but “What level of risk can we accept, and what controls do we need to meet that level?” Companies that deal with governance early on make systems that can grow. Organizations that put off governance rarely get past the pilot stage.
After setting business goals and governance, CIOs need to pick the right architectural pattern. GenAI doesn’t work when businesses try to fit every use case into the same box. The architecture must fit the problem instead.AI copilots help workers by making drafts, getting information, or summarizing conversations. Humans make the final decisions. These are great for helping customers, writing content, coding, and sharing knowledge within a company. AI agents can handle multi-step workflows on their own. They plan what to do, call tools, update systems, and finish tasks from start to finish. This makes them useful for structured, repeatable tasks like changing reservations, adjusting loyalty points, or routing requests. Finally, insight and analytics engines turn unstructured data like transcripts, surveys, and emails into trends and useful information. In big companies, all three patterns work at the same time: copilots give power to front-line teams, agents automate boring tasks, and analytics engines show patterns on a large scale. GenAI doesn’t grow by finding one perfect pattern; instead, it grows by using the right pattern for each problem.
Even with good architecture, augmented work has a bigger effect today than fully autonomous systems. Instead of asking, “Which jobs will AI take over?” CIOs should ask, “How can AI help our people do their best work?” In customer experience settings, GenAI copilots provide real-time policy lookups, sentiment cues, conversation summaries, and advice on what to do next. These features cut down on handle time, make things more accurate, make customers happier, and lower the risk of burnout. Agents have more time to solve important problems and less time doing the same things over and over. For planning this change, a three-tier adoption model works well. AI suggests actions at the Assist stage, and people make the final choice. This is a quick and low-risk way to build trust and show value. At the Automate stage, AI takes care of normal tasks and people take care of unusual ones. At the Autopilot stage, AI does tasks from start to finish while people watch. Most businesses should start at the Assist level, improve their governance, and then move up as they gain more trust and dependability. People who go straight to independence often get stuck.
CIOs need to be very careful when measuring value in order to scale GenAI. A good pilot is more than just a working model; it’s a system that is safe, reliable, and cost-effective, and it can handle real-world conditions. CIOs should look at how much better handle time, satisfaction, resolution rates, cost per interaction, and accuracy have gotten.They also need to look at where the system goes wrong, how often people ignore AI suggestions, how well the model works when there is a lot of traffic, and how it works in unusual situations. Adoption is just as important, do employees really use the system? At the portfolio level, CIOs need to know which projects will have the biggest effect on the business based on how hard they are and how ready they are. Start with initiatives that have a big impact but are not too hard or too easy. These will build trust and give you early wins. Projects that don’t have a clear return on investment (ROI), strong executive support, or clear graduation criteria should be stopped right away. Nothing slows down a company’s progress faster than pilots that never end.
To scale GenAI well, you need a strategic roadmap instead of doing a lot of random tests. The best journeys have phases that overlap. The first phase is all about getting the executives on the same page. This means making sure that every leader knows what GenAI can actually do and setting rules for governance, workforce adoption, and ethics. The next step is to figure out the organization’s most important use cases by looking at them through the lens of impact versus complexity. This makes sure that problem statements are clear, results are measurable, and business owners are involved. Next come governance foundations, where teams set up controls for safety, truthfulness, privacy, and auditability, as well as monitoring and risk management. At the same time, CIOs need to create shared GenAI platforms like model-serving infrastructure, retrieval pipelines, API gateways, and evaluation frameworks that will help with future work. As RAG pipelines and data integrations get better, the first group of pilots should be put through a lot of testing against graduation criteria that decide which ones can grow and which ones should stop. Organizations should only expand GenAI to other parts of the business, set up a center of excellence, and adopt an operating model for continuous innovation after these foundations are in place. The key is to find a balance between quick wins and building long-term skills. Both are important for getting out of pilot purgatory and creating lasting value for the business.
In the end, scaling GenAI is not a technical problem; it’s a leadership problem. CIOs need to be product managers who care about results, architects who design reusable patterns, risk managers who make sure there is governance, and change agents who help the workforce through change. The best CIOs don’t try to get every new model release. They put money into platforms instead of one-time projects, keep track of everything, end experiments that aren’t working early, and put as much money into training and hiring as they do into infrastructure. Most importantly, they know that GenAI is not just a short-term project, but a long-term capability that will change every part of the business over the next ten years. The question is no longer whether companies will use GenAI on a large scale, but which ones will lead the way and which ones will be hurt by competitors who move faster.
About the Writer:
Nitin Kumar is the Director of Data Science (GenAI) at Marriott International, where he leads large-scale AI and Generative AI initiatives across 30+ global brands. His work focuses on building production-grade AI solutions for customer experience, marketing, and service optimization, with a strong emphasis on governance, safety, and real-world operational impact.
He is a frequent speaker at industry and academic forums, including Ai4, MLOps, CDAO events, and university programs, and is recognized for translating complex AI concepts into practical strategies that drive measurable business outcomes.