How to navigate your tech team through the generative AI hype

In the past six months, AI, especially generative AI, has moved into the mainstream with OpenAI’s launch of ChatGPT and DALL-E to the general public. For the first time, anyone with an internet connection can interact with an AI that feels smart and useful – not just a cool prototype that’s interesting.

With this elevation of AI from sci-fi toy to real-life tool, a mixture of widely publicized concerns has arisen (should we pause AI experiments?) and excitement (four-day work week!). Behind closed doors, software companies are struggling to get AI into their products, and tech leaders are already feeling the pressure of higher expectations from the boardroom and customers.

As a technical leader, you must prepare for the increasing demands placed on your team and make the most of new technological advancements to outsmart your competition. Following the strategies below will set you and your team up for success.

Channel ideas into realistic projects

Generative AI is approaching the peak of inflated expectations The Gartner hype cycle. Ideas start to flow. Your colleagues and the board will come to you with new projects that they see as opportunities to ride the AI ​​wave.

When people think big about what’s possible and how technology can enable them, that’s a great thing for engineering! But here comes the hard part. Many ideas that come to your desk will be accompanied by a Howwhich may not be anchored in reality.

It can be assumed that you can just plug a model of OpenAI into your application and, presto, high-performance automation. However, if you use the How and grab the What of the idea, you may discover realistic projects with strong stakeholder support. Skeptics who previously doubted that automation was feasible for some tasks may now be willing to consider new possibilities, regardless of the underlying tool you choose to use.

Opportunities and challenges of generative AI

The newfangled AI that captures the headlines is very good at quickly generating text, code and images. For some applications, the potential time savings for people is huge. Yet it also has some serious weaknesses compared to existing technologies. Consider ChatGPT as an example:

• ChatGPT has no concept of ‘confidence level’. It offers no way to differentiate between when there is a lot of evidence to support its statements and when it makes a best guess based on word associations. If that best guess is factually incorrect, it still sounds surprisingly realistic, making mistakes in ChatGPT even more dangerous.
• ChatGPT has no access to ‘live’ information. It can’t even tell you anything about the past few months.
• ChatGPT is ignorant of domain specific terminology and concepts that are not publicly available to scrape from the internet. It can associate your internal company’s project names and acronyms with unrelated concepts from obscure corners of the internet.

But technology has answers:

• Bayesian machine learning (ML) models (and many classical statistical tools) contain confidence bounds for reasoning about the probability of error.
• Modern streaming architectures allow data to be processed with very low latency, whether updating information retrieval systems or machine learning models.
• GPT models (and other pre-trained models from sources like HuggingFace) can be “fine-tuned” with domain-specific examples. This can dramatically improve results, but it also takes time and effort to compile a meaningful dataset for tuning.

As a technical leader, you know your business and how to meet requirements from your stakeholders. What you need next, if you don’t already have it, is confidence in evaluating which tool is right for those requirements. ML tools, which include a range of techniques from simple regression models to the large language models (LLMs) behind the latest “AI” buzz, should now be options in that toolbox that you can confidently evaluate.

Evaluate potential machine learning projects

Not every tech organization needs a team dedicated to ML or data science. But it won’t be long before every tech organization needs someone who can cut through the crowds and articulate what ML can and can’t do for their business. That judgment comes from experience working on successful and failed data projects. If you can’t name this person on your team, I suggest you find them!

In the meantime, as you talk to stakeholders and set expectations for their dream projects, go through this checklist:

Has a simpler approach, as a rule-based algorithm, already tried for this problem? What specifically hasn’t that simpler approach achieved that ML could?

It’s tempting to think that a “smart” algorithm will solve a problem better and with less effort than a dozen “if” statements handcrafted by interviewing a domain expert. That’s almost certainly not the case when you consider the overhead of maintaining a learned model in production. When a rules-based approach is unmanageable or prohibitively expensive, it’s time to seriously consider ML.

Can a human give several specific examples of what a successful ML algorithm would yield?

If a stakeholder hopes to find some vague “insights” or “anomalies” in a dataset, but can’t provide specific examples, that’s a red flag. Any data scientist can discover statistical outliers, but don’t expect them to be useful.

Is high-quality data readily available?

Garbage in, garbage out, as they say. Data hygiene and data architecture projects can be prerequisites for an ML project.

Is there an analogous problem with a documented ML solution?

If not, it doesn’t mean ML can’t help, but you should be prepared for a longer research cycle, requiring deeper ML expertise in the team and the potential for eventual failure.

Is ‘good enough’ precisely defined?

For most use cases, an ML model can never be 100% accurate. Without clear guidelines to the contrary, an engineering team can easily waste time trying to get closer to the elusive 100%, with each percentage point of improvement taking longer than the last.

Finally

Start evaluating any proposal to put a new ML model into production with a healthy dose of skepticism, just as you would a proposal to add a new datastore to your production stack. Effective gatekeeping ensures that ML becomes a useful tool in your team’s repertoire, not something stakeholders consider a boondoggle.

The dreaded Trough of Disillusionment of the Hype Cycle is inevitable. However, its depth is determined by the expectations you set and the value you deliver. Channel new ideas from across your company into real-world projects — with or without AI — and improve your team so you can quickly identify and capitalize on the new opportunities ML offers.

Stephen Kappel is head of data Code Climate.