29 January, 24
Tom Kerswill, CTO, Cloud Bridge
Over the past year, anybody who's remotely technical will have found it hard to avoid talk of Generative AI - especially in the form of Large Language Models (LLMS), such as ChatGPT. As an AWS consultancy, we've done a great deal of experimentation with Amazon's new framework for generative AI foundation models - Amazon Bedrock.
As with any technology that's progressed quickly along the hype cycle, LLMs have become the multi-tool, the answer to any problem. So much so, that people want to use them without really knowing what problem they're trying to solve.
We've had large numbers of pre-sales calls with customers who are excited to apply GPT-style technologies to problems in their business; but those conversations end up being about the organisation's data.
Conversations about data are a good thing, and we've ended up building some fantastic projects for clients - but they haven't always ended up using generative AI.
And there's a recurring theme. Most organisations have data across many different systems, and it's hard to join that all up to get meaningful insights. LLMs gave a glimmer of hope here. The dream is a chatbot which is given access to all a company's data. You just ask it any question and get a considered response back - full of magical insights that came from your data.
It seems reasonable. After all, Chat GPT was trained on a large subset of the Internet and is thus able to synthesise responses that do a good job of mimicking human knowledge.
But giving a chatbot access to your company's data is very different from training an LLM. To do the latter, you need millions of pounds, and access to huge compute power. AWS can help here, if that's what you want to do. But for most companies that's a sledgehammer to crack a nut. Especially given that if you want to retrain the model in a year's time, you'll have to do that all again. Most people leave this art to the big players - Anthropic, Meta, OpenAi, and Amazon themselves, with their Titan model. And then build something on top - using Amazon Bedrock, for example. 
So when we talk about giving a chatbot access to a company's data, we almost never mean training a model on that data. Instead, we take a pre-trained model and add something called 'retrieval augmented generation' (RAG). This is a bit of a hack. We're starting from an LLM that's already been trained on a massive amount of generic data which makes it excellent at answering general queries, but the model knows nothing of your company's internal data.
There are relatively few ways we can give the model data. It's a black box, with a few levers, such as temperature and Top P, which affect how the LLM produces the next word. The only other input we have. is the prompt itself. RAG simply stitches the relevant data ('context') at the end of the user-supplied prompt. 
What does 'relevant data' mean ? Well, we can't give all the company's information to the LLM every time we ask it something. Instead, we tend to store that in a database (typically, a 'vector database') and do a search for data that looks relevant, then pass some of that through to the LLM when we ask it something.
That's what new tools like Amazon Q for business do. They use databases, such as Amazon Kendra, to find the most relevant data. Typically a multidimensional 'vector' is used. That's a load of different numbers whose length and 'direction' can be calculated (the numbers themselves come from words in the data, via 'embeddings' - you can think of this as capturing the 'meanings' of words). By comparing the vector of each bit of data, with the prompt itself, you can get an idea of data that looks similar.
We've done loads of experimentation with Amazon Q and other chatbot-type systems that use RAG to access your private data. It's understandable that people are getting excited about this. Imagine being able to ask the chatbot anything and receive an answer that takes into account every bit of information your business holds.
Except that's not what's happening. Instead, RAG is doing a fairly crude search of your data and just pasting that into your prompt. The results are variable.
If I'm after a specific bit of information, I don't want to have to talk to a chatbot, and have my prompts used to search a database in the background. What do I want instead? A precise, consistent search engine which allows me to type what I want - using natural language, so it will 'know what I mean' even if my search terms don't appear exactly the way I typed them.
Users have become pretty good at searching data. I think what people really want is a 'Google for company data.'
I've come to realise from these GenAI experiments, that I get much better search results by cutting out the middleman (eg, a chatbot with a tendency to hyperbole, which sometimes gives mixed results), and simply going straight to the database.
There's class of tools that excels at this - Enterprise Search. And lo and behold, Amazon Kendra, which is what Amazon Q uses underneath, is an excellent enterprise search tool in its own right.
I've concluded you're better off going straight to the source of the data and simply asking it what you want. You can then use some lovely advanced filters - a bit like searching Amazon or similar - to narrow down to exactly the result you need.
The tool is simple, but the tricky bit is giving it access to your data underneath. That's where you need a consultancy that understands AWS and data integration.
We've built out tooling that ingests data from some common SaaS products, as well as mainstream databases. And all this is ingensted into Kendra. Some of this is based on Kendra's own connectors. We've also built our own secret sauce on top; and we're set up to be able to build new connectors based on the unique tools our clients are using. But the most important success factor is the consultancy we provide on top, and the ongoing support we can offer once we launch your enterprise search tool.
This stuff is hard and it's still changing. By holding initial workshops, we're able to work with you to ask the right questions and come up with a solution you need. And one that'll give value for years to come. Too often, we're seeing people starting with the answer - which often ends up being 'use generative AI'. We like to start with the questions, set aside the hype, and come up with something that gives you business value as quickly as possible.
