Welcome to the KuppingerCole Analyst Chat. I'm your host. My name is Matthias Reinwarth. I'm analyst and advisor with KuppingerCole Analysts. And my guest is Alexei Balaganski. He is the CTO of KuppingerCole, and he is a cybersecurity lead analyst. Good to have you here today, Alexei. Welcome.
Hello, Matthias, and thanks for having me again. But today you are here not in your role as the cybersecurity advisor, at least only partially. I think you are a bit more the CTO for our full audience. We want to talk about artificial intelligence, about generative AI, and how to properly use it. So kind of a primer, a step back to say, OK, we are all using it. Are we doing it right? Exactly.
Like we've been discussing so many sophisticated topics and highbrow analysis and stuff like that, but we are constantly forgetting about the basics, the hygiene of AI users, because if we have not figured out those basics for everybody within your company, nothing else will work because you have to start from level 101, if you will. And this is exactly what we are going to discuss today. Right.
So yeah, the first step back is we want to talk about generative AI. Let's look at the term generative later. Artificial.
OK, we have a feeling for that. Intelligence. That's the important part. What do we consider to be intelligence and does this help in approaching this topic?
Well, this is actually a huge discussion on its own. It's a philosophical subject which has been discussed for centuries, way before computers, if you will. Because to be honest, we don't really know what human intelligence actually is and how it all works. There are of course multiple assumptions and theories. Is a human brain, for example, really just a biological computer? Or is there something like a soul, if you will, in there which cannot be replicated? We don't know. And there is absolutely no way we could even tackle this topic in a podcast.
But one thing we have to communicate to all of our listeners and readers is that, oh, don't trust the term. When you are working with artificial intelligence, do not consider it like an artificial person, an artificial expert, an artificial colleague, agent, helper or anything else. Artificial intelligence is still just basically sophisticated mathematics running on a very powerful computer hardware. And it does not think, it does not feel, it does not know much in the same way we humans do. And we cannot really even compare an animal to a very nice dog or dolphin or even a colony of ants.
We don't really know. The closest thing you probably find as an analogy for modern generative AI is those proverbial monkeys with typewriters. The only difference is that we are talking literally about billions of those. Because as you know, a modern typical LLM has billions or tens of billions of parameters. And those are exactly the monkeys. The question is, how should we treat that crazy zoo on steroids to actually get something useful out of it?
But in the end, the results that people get out of that, even when just asking a simple question, and we get to that later, the results are convincing. They are good. They are fluent. They really give you the impression that there is somebody sitting on the other side of the line that is talking to you and really providing proper answers. And even if it's wrong, even if it's incomplete, even if it did not get the question as you've meant them, because you were not really precise, there are still results that at first sight look brilliant.
So why are we in a situation where we can get results from simple prompts that are convincing and wrong? By the way, what you have just described is essentially the Turing test, right? This is what Alan Turing came up with in the fifties or even late forties, when he said basically an AI has to be convincing enough so that when you chat to it, you will never even understand that you are chatting to the computer, not human. In that sense, of course, modern LLMs have way bigger capabilities and are way more convincing than a typical chatbot or script from those times.
Although you have to admit that chatbots actually existed way before LLMs. They were not particularly convincing, but they were chatty, if you will, and they could answer your questions. But nobody called those artificial intelligence. So what is the difference now?
One thing, again, you have to understand that Turing did not actually mean it as a scientific, proper way to test the capabilities of artificial intelligence. He was basically joking, if you will. It was a sarcastic response to people back then in the forties who did not believe in the future of computing. It's actually the same story as with the probable Schrodinger's cat.
No, Erwin Schrodinger never meant his cat experiment to be a real description of quantum mechanics. It was a joke. But too many people just don't get it.
So again, if you are communicating to an LLM, a chatbot or an agent, and it sounds convincing enough, this is not the reason to believe that it's intelligent. It can still be useful.
But again, to not believe it anymore, you would believe a stranger in the street. As you said, don't believe more than a stranger in the street. But nevertheless, we do it all the time. It has arrived at the desktop of almost everybody who is working in a professional environment, who is using a chat application like WhatsApp. So these technologies have arrived and everybody is able to use them.
So maybe the main takeaway for our audience today is to get a bit better in using this, especially when it comes to corporate environments your real life business where you're using these technologies, be it sanctioned or unsanctioned. I don't want to discuss that. But really to say, okay, how can we get better? What is definitely wrong? What is the biggest mistake one can do to use an AI in any context?
Well, again, AI, well, let's continue using the term, but again, the most important takeaway for everybody should be that AI is not a thing. It's a spectrum from a simple script to a sophisticated, well-empowered agent. This is all AI and AIs are very different. And they are just tools which can be useful, but which can be dangerous. They can be very efficient, but they can also be very costly. They harm our environment and they also definitely harm our social interactions, if you will. We've already seen that.
But again, when you're using them properly, when you're using them for work, they can definitely bring you a lot of useful stuff. And key things here are basically to stick to a set of rules and very simple principles. And if we agree to talk about good AI usage culture as the premise for our discussion today, then there are obviously some things you have to stick to, do, and some which you have to avoid at all costs. Let's kind of go through all those things and discuss. So one obvious, probably the most important do of good AI culture is that you have to disclose AI usage, right?
Sounds pretty obvious, but a lot of people just don't get it. You have to understand that AI does not replace you as a person with all your responsibilities and so on. Even if you're using AI to write a report or create a music track or a PowerPoint presentation, it's still you behind it. You have to be responsible for every consequences. And of course, you have to communicate clearly that it has been created with an AI support.
Obviously, you can do this whole thing completely differently. You just start using AI tools as a starting point, brainstorming, drafting, analyzing external sources and stuff like that.
Basically, you feed tons of information you would never be able to read it through years or months at least to those crazy monkeys, typewriters. And they will summarize all the stuff down for some digestible form and size. And you can use it in your work normally, just like you would use a calculator or Google or something like that, or Wikipedia. This is kind of a different approach. This is AI-assisted content generation, if you will.
But again, you are still the author, you are still responsible for good and bad things in your content, but kind of AI is just one tool in your toolkit. Obviously, you should never trust anything AI generates. We all know, again, this has been discussed today, that AI large language models are probabilistic by nature. They are unable to give you a repeatable, reproducible, concrete answer to any questions. Every time you ask something, the result will be different. And there is a strong possibility that the result will just be incorrect by nature because they hallucinate all the time.
So you have to validate everything that AI generates. Again, this sounds completely self-explanatory and evident to any sensible person, but it's so convenient, it's so easy to just be lazy and copy-paste whatever a chat GPT tells you into an email or a report.
Obviously, this is the biggest no-no and the biggest don't. And again, for obvious reasons, because you are still responsible. But you've mentioned that already. You said providing tons of information. That is context. So providing the right context, ideally context that is not just taken from anywhere, stolen, or just copied from somewhere, but really very well curated and understood that this is a basis that can create the results that you actually want. So providing context information, that is one of the important starting points.
So it does not have a chance to go left and right and just hallucinate or take wrong sources. Right. We will get to this in more detail just in a minute. But before, if you want to finish with the do's and don'ts, obviously, you need to have as much of that content and context as possible. Because again, do not assume that AI knows everything or that AIs understand some unspoken conventions within your company or in your society or just, well, in the world, because they don't.
Again, because of AI, AIs of today are just huge mathematical models running on computers. They only have bits of information, but they have nothing to assume if they don't have that information. This is why they hallucinate, because they do not understand, they do not know anything. They will try to invent data out of thin air. This is what we interpret as hallucinations. So obviously, you have to give an AI as much content and context for the job as possible. But on the other hand, this is so tempting to just give AI an unlimited access to everything your company has. All your data.
This is, of course, the worst mistake we know people are doing right now, because we hear so much about data breaches and data leaks and disastrous, incorrect decisions made upon this data, which has been mindlessly copied and pasted into especially a publicly operating large-language model. This is like the first absolute don't, if you will, of any AI usage. Every company has to have a set of AI usage policies, which clearly instruct that you cannot and should not ever leak sensitive company information to a third-party model.
There are solutions which tackle that, SLAs and special contract clauses running AI models on-prem and so on. But again, this is something which technical people and legal people would have to have set up specifically for you as a business person beforehand. This is not something which a normal non-technical person can ever address themselves. So basically, again, when using AI, follow the company policies. This is an absolute must to avoid catastrophes. Right. You've mentioned the do's and don'ts. Are there any additional do's?
Before we go into more detail that you would like to highlight, you've mentioned attributions, so being transparent that you're using AI. What else is there that you would like to highlight?
Well, obviously, even though AI agents are now kind of exploding in popularity, especially with all these claw bots and whatever they're called nowadays, the things which you can run locally and let them kind of act on your behalf, this is still extremely dangerous. This is still highly untested. If you want to be an alpha or pre-alpha tester for this emerging technology, you're welcome, but never use it with company data. And even if you're doing it at home, I would rather recommend you to set up a separate computer in a separate network for that.
A lot of people are talking about buying like a Mac mini. Fine, that's a great idea. Just make sure you do not connect that agent to your work email account or SharePoint server or anything like that. Because again, we have dozens if not hundreds of reports, those AI agents breaking stuff on a massive scale. Because they are, again, they are still in the very infancy of that technology. There are no guardrails, there are no security frameworks, there is very little governance and access management. So the consequences can be very unpleasant. So if you want to experiment with it, feel free.
That's always rewarding and interesting. Just don't use it with your company data. And I think also that these open ecosystems that we now have with this AI, just because you're using, I always say AI as the generic term for that, but once you use such a tool, you are not obviously sure which sources, which resources are actually accessed in the background, what it's actually using. There are these open protocols, there are these open communication mechanisms behind that. There are restricted ones that we can or should use if we need to do it.
But when we use a public service or such an agent, you're not really sure what happens in the background once you give away your API key to have access to this cloud, open AI, whatever out there. Right.
Well, you know, an open ecosystem always sounds fun until you realize that there will be some very grumpy people coming to you saying words like software supply chain security and third-party risks and again data breaches and GDPR fights and whatnot. So yes, unfortunately, there is already a very popular open ecosystem for those personal AI agents, but there's very little governance and security. You will never know what that really interesting plugin would actually do.
It would help you plan your vacation or it would mine crypto coins on your hardware or it would steal all your family photos, for example. You would never know.
So yes, we have already said a lot of don'ts. Perhaps we should kind of move on to some useful positive recommendations.
And yes, one of those topics which I hear people talk about a lot is, of course, prompt engineering, because somehow it has been established that nowadays you don't have to be like a programmer or a data scientist or like any kind of IT person, just have to learn prompt engineering, whatever it means. And to me, it always sounds like some dark science or magic, but it's actually not. And consider it a hot take on my side, but I think that prompt engineering basics can be learned in like 15 minutes.
And all the rest on top of that, it's just, you know, nitpicking and fine points which don't actually bring you a lot of return. And of course, some of those recommendations for prompt engineering are also coming full of do's and don'ts. And one of the biggest don'ts for prompt engineering is like, do not make it too complicated.
Again, the whole appeal and premise of LLM so that you can talk to them. Like you would never imagine going to your colleague in HR or finance and bringing in a script with 10 pages to ask a question, right? You don't have to do the same thing for LLM. Just ask a question, read the response, ask another question, and well, engage an LLM in a conversation. This is how you've been doing this all your life with humans. There isn't a lot of difference with LLMs. You just have to remember that, again, an LLM is not as smart and intuitively understanding as a human.
So we just have to stick to a list of very simple and sensible rules. Well, the first rule is obviously do not expect it to understand what you are thinking about. It will only understand what you are writing and asking about. So do not tell something like, hey, chatGPT, make me a financial report.
Instead, write something like, hey, chatGPT, write a 500-word report on the financial results in our customer database across the European and North American regions. Again, the more context you will give it, the better the result will be. We just discussed it earlier, right?
Again, provide context in the sense of giving it inputs or chatGPT now supports a lot of integrations with your existing business tools. You can just connect your Dropbox or SharePoint site or OneDrive. You don't have to upload PDF documents all the time.
But again, if you are giving it permanent access to your business resources, you have to think about access management. Whatever chatGPT can access, everybody else in your company will access as well. Because again, chatGPT doesn't discriminate who is talking to it now. Is it a CFO or an intern? This is why you have to discuss it with all your security and IT and legal teams first. But as you have this figured out and set up, just feel free to limit business context as much as possible. Another thing which a lot of people tend to somehow miss is that modern MLMs have a context limit.
They can only keep so much data in its memory buffer before they forget. Yes, more recent models have increased this context into dramatically, but apparently it doesn't actually help. This is why it's a very bad idea to just open one chat and ask 50 questions in the same chat. It will sooner or later pollute the responses and it will increase the chance of hallucination extremely. This is why kind of a rule of thumb should be one chat for each job or even each task.
If you are writing two reports, just open two different chats in different browser types, if you can better create a different project for each job and create related but different chats within those projects. Another thing that a lot of people forget is that chatGPTs of this world are great at imitating different kinds of people, different personas. They can be your friendly drinking buddy or your strict teacher or a mentor or a financial analyst or whatnot, but you have to tell them.
Basically, you have to explain what kind of persona they should be acting as, what should be their writing style, should they ask you a lot of questions, if they miss something or not, and so on. The better you will set it up for you, the better it will work, the better it will support you in your daily work. So maybe invest in a couple of hours on doing this once and then kind of reuse those existing personas will help increase your productivity a lot. And of course, a prompt doesn't have to be a 10 pages document. Start small and then try to refine the outputs.
Feel free to engage in a conversation with an LLM, because conversation is the best way to kind of fine tune your expectations and improve the quality of the responses. And finally, you should never just take the first response from chatGPT and consider it true, even if it's something simple. We know that you can ask chatGPT how much is two plus two, it will happily tell you five, just because something was happening in the backend. Always verify, never trust. If you see something which sounds or looks suspiciously, ask for clarification and explanation.
By the way, this is another topic which kind of modern chatGPT are improving upon, but they're still kind of lacking it on a human level. It's explaining how they have reached their results. You can already do it to an extent with chatGPT and other tools, but you have to practically look for those. They will show you links to specific sources, they will pop up a tooltip saying like, I am thinking about this or I'm considering that, but you have to practically look for those hints.
Maybe in the future, they will come with a better, more readable and understandable protocol and look for later audits. But right now, you have to take very specific care of understanding how this specific result was achieved. Because again, the opportunity for a catastrophic mistake is very tangible. There are multiple reports on the internet about companies doing some strategic decisions based on incorrect inputs from chatGPT, if you will, or other LLNs.
And those incorrect inputs were the result of, again, hallucinations, insufficient context, access to the wrong data, or just people being too uncritical. They would see a number which sounds plausible and nice and never care to verify and get a second opinion. Are we really getting as many customers in this region? Are we really expecting to build a new office or in the street and whatever? Are we getting enough feedback from whatever contractor?
Well, you cannot trust chatGPT. You have to ask the actual contractor or the actual customer. Do it occasionally from time to time.
Again, zero trust for AI. This should be your motto. I don't know how often I've said in these 200 something episodes of this podcast, apply a risk-based approach. The higher the potential risk status behind it, the more scrutiny you should apply from simple sanity checks to really full-scale verification by a human, by another AI, by five other AIs, just to make sure that you have a basic level of trust into the results. As you said, zero trust towards AI is a good approach.
But maybe on the other hand, if things work for you and they deliver good results and you've came to that in an hour of work with such a prompt, make sure that you keep the results. It is not promised that they will work the same way the next time, but there are chances that they do and at least can act as a good basis. You don't have to build your own huge prompt library, but to have something just close to your desktop or even in a note app where you can say, okay, this prompt worked for me. This should really speed up and improve your overall experiences and the results.
Any other recommendations from your side, apart from role setting, as you said, providing context, what else is important? Well, I would probably expand on your recommendation a little bit and I would say you have to document your work with AI. Documenting this work doesn't just mean keeping all the logs on the chatGPT website, for example, but actually writing down the good ones and the bad ones, and you have to document not just your prompt, but also which model were you using? Were you providing one PDF document as context or 50?
Did you use an MCP server or did you involve some kind of external tool or agent to be in the loop, if you will?
If you document all this, you're basically acting like an AI scientist, if you will, and you can always share your findings with your colleagues or with your management, because this is exactly the kind of added value which can be derived from another mundane, everyday AI work, if you will, because if you can improve the performance of your entire company by 5%, let's hope it will be rewarded accordingly, or if you can prevent your colleagues from making a catastrophic mistake, that would be even more important.
So, yes, it is absolutely sensible to not just repeat the same steps you are doing just because you've learned it once a year ago. Grow your human knowledge along with AI technology as well. The AI evolves on a weekly basis. You have to keep up with those developments, at least on the very superficial level, understanding what's now possible, which wasn't yesterday, what works differently in a new tool or a new model, and so on.
And again, kind of documenting those findings will help a lot. Right.
So, before we get to the end of this episode, we've mentioned the do's, the don'ts, but I think there should also be a mindset, a spirit towards AI within an organization, so a kind of good AI usage culture. You as a CTO, are there recommendations that you are using that you really have the benefits from that? On the other hand, it's also to the benefit of the company and maybe also to the group, to the company as a whole?
Well, I have to confess, I have always considered myself not outright an AI disbeliever, but at least a skeptic. Maybe because I started learning that technology way before it kind of exploded in popularity, or maybe I was just kind of looking too deeply into the limitations, but I always thought, surely I don't need AI in my daily work, because I have been doing this for 30 years without it.
But no, I'm absolutely using it on a daily basis, if only for small things. For example, when I'm writing a blog post, and I'm just stuck, I have a writer's block, I just don't know, I'm looking at the blank page for hours, I will just turn to change, and say, hey, give me a couple of ideas. And sometimes those ideas are ridiculously stupid. They will give me, well, my own idea and break my writer's block.
Again, you should absolutely be using and experimenting and trying all the different new things. Just keep in mind that you have to do it responsibly, not invest too much effort and resources, and if you see that it doesn't work, just put it aside and move to something else. But more importantly, again, keep in mind this balance of risk and reward. AI is still very expensive, and this can still lead to catastrophic issues if used incorrectly.
But again, if you have identified 5, 10, 50 small use cases or potential applications, great. Experiment, share, and again, keep in mind that it has to be done responsibly, so do not violate company policies. Think about the environment and costs, if you will.
And again, treat this as another tool, because not everything has to be done with AI. This is probably the biggest wrong we are doing with it nowadays, because of all the hype. We are trying to shoehorn AI everywhere, even in places where a simple script or whatever, a small application would be sufficient. Think outside of the box. Try to imagine the use cases and return on investment in those use cases, and apply AI whenever you see opportunities. But think about the risks.
Especially in our business, as analysts, I think the term original content, which is not fabricated by a generative AI, but is the result of your research, your brain, your reasoning, your out-of-the-box thinking, I think this will increasingly get more important. And I have seen that in guidelines for journalists already, where the publishers really demand for non-AI content. So use it where it makes sense, use it where it really benefits, where it speeds you up, without influencing the content, especially negatively.
And all of us really, today, realize content that has been created by generative AI, because you understand several kinds of phrasings that are immediately hinting at AI, and it's simple. Let's be honest. Garbage in, garbage out. It was true before AI.
Yes, it's even more true nowadays. Whenever you see all those obvious signs of using chatGPT, for example, it's not because the author was using chatGPT, because it only used it, and did nothing else on top. And this is the biggest sin, if you will, of using AI in your work.
Again, you as an author, as a worker, the creator, if you will, you are responsible for your outputs. Whether you have made them with chatGPT, or assisted with chatGPT, or completely manually, or you used, I don't know, like a chisel to cut it, or install, doesn't really matter. The only thing that matters is quality. Your opinions, your ideas, your numbers, if you are in that kind of work, and you have to stand behind every one of those things. As long as you can do it, it doesn't really matter which tools you have been using, right? It's all about creativity and responsibility.
If we can agree to all be, just as you said, skeptic, but an optimistic skeptic, when it comes to using these technologies, I think we have already achieved quite a bit, so that we understand the technology, that we understand where the limitations are, where the challenges are, when it comes to legal, to compliance, to reputation, when it comes to cyber security incidents that could happen. I think that then we are in a better place, and we are still able to openly use these technologies for the benefit of ourselves or organizations, but with a skeptic approach, which I think might help.
Before we close down, any final recommendations where you would say, oh, this is the next good thing, or this is the next threat that we should look at? Well, there are so many nowadays.
I mean, Matthias, you are a musician, right? So you probably feel it even more. There are some professions or some hobbies, if you will, that will suffer a lot from the consequences of AI. And we already see that there will be even more. I have always been saying that if your kind of work can be automated, then it's your bad decision of choosing this line of work.
So yes, you will be replaced by AI. But nowadays, we feel that kind of the AI is stepping into territories which were never supposed to be automated, which again, creating music, or making movies, or writing analytical papers about cyber security, if you will. So we are all in danger of being replaced by AI. But only if the AI can be as good as our human-powered outputs, right?
So far, it's not there yet. And I guess it's our primary responsibility to make sure that we keep up in our quality, in our expertise, and opinions, and whatnot. This is what will matter all the time. Because I would imagine in 50 years, there will be a lot of AI-generated music everywhere. But there still will be people attending other people's playing, right? Concerts. The same would apply everywhere in business, whether it's about cyber security analytics, or finance, or marketing, or whatnot.
There will be AI tools, but the decisions and the creative breakthroughs and the biggest risks have always been the prerogative of a human. So let's keep it that way. I'll leave it with that.
Great, great final words for today. Thank you, Alexey, for being my guest today, for sharing your insights, for just having this very basic, but really, really useful episode together. A bit off-kilter of what we usually do, but I think this is really helpful. And maybe it inspired one or the other to rethink their approach towards using these technologies, which are great, which are at hand, and which need to be used properly. So thanks again, Alexey, for being my guest today. Thank you.
And again, I am not AI. Please remember that.
At least, but the AI would say the same. But hey, so see you next time, or your agent. See you. Bye-bye.
