Small generative AIs: size matters

Today, I want to talk about small generative AIs – the little powerhouses that are making a big impact in the world of software development.

Now, I know what you might be thinking: “Isn’t bigger always better?” Well, not exactly. When it comes to AI models, bigger usually means more accurate and impressive results. But (there is always a but) the amount of data also brings with it some limitations and risks.

We've all heard about these things called large language models (LLMs) – for example, OpenAI's ChatGPT, Google's LaMDA and BARD, Hugging Face's HuggingChat, Meta's LLaMA, and Amazon's Titan Text. It's amazing how quickly companies jumped on the LLM bandwagon after the smash success of OpenAI.

These models are impressive (at least the ones that are publicly available) and could not exist without millions of gigabytes of data. No wonder AI products are using the number of parameters (in the billions) as a reference, like console manufacturers used to talk about bits in the 90s and early 2000s.

But that size comes at a cost. The server architecture for this type of product is huge, and this is just the tip of the iceberg: data collection and cleansing, engineering costs, time investment, and more. This isn't the kind of product you whip up in your garage over a weekend.

And that's without taking into account some of the ethical issues surrounding data collection, data mining, and other topics that have landed OpenAI in hot water in recent months, like the Italian ban. And believe me, we'll get to that, but before looking at an even bigger issue with these models, the best way to explain it is by talking about a well-known classical figure: good old Socrates.

Socrates' Woes: Episteme vs. Doxa

Socrates is an enigmatic but significant figure of the classical period; almost everything we know about him is told through the eyes of his supposed disciple, Plato. For some philosophers, Socrates was not even a real person, but rather a literary device to explore Plato's philosophy through dialogue.

Leaving that debate aside, Socrates was famous for being an obstinate man who extracted answers from everyone who dared to have an opinion. The Socratic method is a painful process in which, through the act of questioning, supposed knowledge is dismantled, giving way to the “truth” behind the subject in question.

He was so fanatical in his search for the truth that he ended up making political enemies that cost him his life. Accused of corrupting young people, he was unjustly sentenced to death. Even when he had the opportunity to escape, he decided to accept his fate, as it was his responsibility as a citizen of Athens.

In other words, Socrates was against doxa, the beliefs and opinions held by a group of people. Instead, he tirelessly pursued episteme, which refers to “true” knowledge. At that time, the way we understood the truth was very different, but nowadays the episteme would be equivalent to knowledge based on facts and evidence.

Here's the sad truth: when we rely too much on grand language models, we run the risk of prioritizing doxa over epistemology. The model may be able to generate a convincing argument or plausible explanation, but that does not necessarily make it true.

The reason is twofold. First, language models don't understand concepts the same way we do (in fact, they don't understand them at all). They are really good at understanding which words should go together, but not why they should go together. This is what leads to hallucinations when AI makes up facts because, statistically, it makes sense for those words to go together.

The second reason is that most of the data in the world is actually doxa in the Greek sense and not episteme. The same data that models are built with. Now, before we talk about the implications of the statement, I want to make something extremely clear: doxa is not the same as lies or falsehoods. Instead, it would be more like a “half truth”. For example, most people know that vinegar keeps food from spoiling, but ask them why this happens and most people probably can't explain it.

In simpler terms, just because a model can talk the talk doesn't mean they can walk the talk. It's like asking my friend who is always spewing conspiracy theories about the moon landing to help me with my history homework. Sure, he might be able to come up with some wild theories about what “really” happened, but that doesn't mean he knows anything about the actual historical facts.

And that's the real danger of relying too much on LLMs. We might be tempted to trust the model's results without questioning whether they are actually accurate or based on solid evidence. It's like letting a Magic 8 Ball make all your business decisions – sure, you might get lucky every now and then, but answer me: would you fly in a plane built entirely with instructions from ChatGPT?

So what is the solution? As with most things in life, it's all about balance. Great language models have their place, but we need to be careful not to let them completely take over. We must always question the result of the model, checking the sources and checking the facts. And if we are unsure about something, we must be willing to admit it and seek additional information.

Now, there is another alternative. How about, instead of relying directly on the LLM, we just use it as a foundation, as a starting point to build specialized AI with carefully selected data? Just as we humans begin our journey in education by learning a little about everything and, as we grow older, we focus on specific areas.

The scope of these focused models is quite small in comparison, but at the same time, their results on specific subjects are much more reliable. It's like adjusting stable diffusion with Velásquez paintings. From that moment on, this particular model will have a harder time making every conceivable type of image, but he will become very good at painting like Velásquez.

Small Generative AIs vs. Large Generative AIs: Pros and Cons

Alright, let's talk about small generative AIs versus large generative AIs. It's like comparing a pocket knife to a machete; Both have their uses, but one may be more appropriate than the other depending on the situation.

Small generative AIs can be like your trusty pocket knife: compact, efficient, and reliable for certain tasks. They are great for generating small snippets of text or images quickly and accurately. You don't need huge computing power to perform tasks with them. Furthermore, they can be easily integrated into existing systems without causing much disruption.

On the other hand, sometimes you need that big, bad, machete-like AI to solve difficult problems. Large-scale models have access to large amounts of data and processing power, allowing them to generate more complex content with greater accuracy. They may not be as easy to handle or integrate as smaller ones, but they are certainly effective when it comes to difficult times.

I've worked with both types of models in my time developing software, and I've found that choosing between small-scale and large-scale AI really depends on what you want from your AI system.

Sometimes all you need is a quick decision-making tool that generates simple code snippets or strings based on some predefined patterns – something relatively mundane that doesn't require heavy computational resources – so small models serve best.

Other times, though (like if you're trying to deepfake), the big guns should come out. Larger models trained on large amounts of data will help us get closer to achieving what we aim for: developing AGI skills, such as creating complete virtual worlds, indistinguishable from reality.

There are also training cost considerations here. Running training involving vast datasets for longer periods of time requires immense resources and therefore wastes unnecessary energy. For small problems, big models are like blowing up a country just to kill a fly. Therefore, it is a tool that must be used wisely.

The future of small generative AIs

First of all, let me just say that I'm super happy with what's coming up. As someone who loves tinkering with technology and pushing boundaries, I can't wait to see how far we'll go with these mini wonders.

My gut feeling is that as processing power becomes even faster and cheaper than it already is, we'll see some truly amazing things happening with small generative AIs. Think about it: right now, they're impressing us left and right by creating stunning pieces of art or music within certain human-defined parameters (like color schemes or chord structures). But what if they had more freedom? What if they were given the creative reins?

I know some people might be worried about this idea. After all, shouldn't we be the ones making art? But hear me out: I think working alongside small generative AIs could lead to some really interesting collaborations between humans and machines. Imagine being able to bounce ideas off your own personal AI assistant until you come up with something truly unique and unexpected.

As AI improves and new models are built from fundamental models, we will see higher quality results from these AI tools. The future isn't just about collecting more data and feeding it to the big fish; it's also about becoming leaner and more efficient, about saving energy, about creating the right tools for our needs.

There are many practical applications for this technology beyond just generating beautiful images or music. Small generative AIs could help automate repetitive coding tasks or even generate entire programs from scratch using data instead of pre-written rules.

Of course, there are also potential downsides to consider when thinking about handing over parts of our creativity (and livelihoods) to machines. It is important for us as software developers to not only explore new technologies but also evaluate their ethical implications on society at large.

Case studies: Small generative AIs in action

When it comes to small generative AIs, there has always been a bit of skepticism in the software development community. Some developers believe that these small algorithms are very limited in scope and cannot provide any significant impact on their work.

But as someone who has actually implemented several small generative AIs in my own projects, I'm here to tell you: size doesn't matter! In fact, some of the most innovative and impactful solutions I've created have come from smaller AI projects.

Take, for example, a project I recently worked on. We were developing an application that required the ability to generate personalized recommendations for users based on their behavior on the platform. We originally considered using a larger pre-trained AI model, but after experimentation we decided to opt for a more compact design that was better suited to our needs – consuming fewer resources while still delivering impressive results.

Some enthusiasts have had great success in building small language models by training them with GPT results, for example GPT4all. (By the way, don't do this, it goes against OpenAI's terms of service.)

To ensure its accuracy remained high, we trained this algorithm extensively, testing different inputs and parameters until we achieved exactly what we needed: fast operation and accurate results delivered consistently across all devices!

While it may not have been as robust or comprehensive as many other existing models, our compact genAI proved invaluable through its reliable recommendations, efficiently leading us toward new feature ideas that improved the user experience without sacrificing the quality or scalability in development deadlines – keeping things leaner than ever in our Agile team!

So if you're skeptical about implementing small generative AIs in your next project because they seem insufficiently equipped, think again! These little guys pack incredible performance where it counts: optimized performance at the lowest cost of processing power!

The Ethics of Small Generative AIs

Let's face it: AI is like a genie that can grant our every wish, but it comes with its own set of terms and conditions. As developers, we have a responsibility to create intelligent systems without compromising privacy or ethics. And when it comes to small generative AIs, the question arises: does size matter?

These days, we see a lot of small AI-powered apps that generate images, music, and even email content. They are useful tools for amateur artists who want to express themselves creatively while learning the craft. But as these applications become more prevalent and powerful, they could potentially raise intellectual property concerns.

For example, you wouldn't want an app that claims ownership over your generated art or idea because it was theoretically programmed by the algorithm on its servers – talk about robotic tampering! Similarly, imagine using an app that generates music tracks based on user input, only to face copyright issues in the future since you've unknowingly crossed lines with already copyrighted pieces!

These ethical dilemmas need to be addressed (and are already being addressed) before such applications become widespread, limiting any future negative impact.

Another area where small generative AIs could run into problems would be their potential susceptibility to exploitation by rogue elements or interest groups with less than pure intentions. Ensuring there are no abuses in certain fields could help prevent automation from replacing jobs too quickly (consider how many people felt uncomfortable when Uber introduced autonomous cars).

I'm not saying that this whole business of smaller but powerful creative ally apps doesn't have fierce potential; however, by the same token, any new invention has parallel possibilities associated with it. We just need to properly balance both sides while striving for responsible practices.

Conclusion: Why Size Matters in Generative AI

The larger the dataset and model in generative AI development, the greater the chances of generating high-quality results. It's like making a cake; The more ingredients you choose and mix, the greater your chances of making a delicious dessert.

But don't let this discourage those who are just starting their journey with AI. You don't have to grow up a lot or go home right away. As they say, “small but mighty” – small generative AIs still have a lot of potential for creative outputs and learning opportunities.

And if there's one thing we developers love to do, it's share our code. Even small models can join forces with other models through federated learning techniques, which allow multiple algorithms in a network to collaboratively train with each other without being merged into a compelling model, incorporating cross-device privacy protection mechanisms such as differential privacy (pun intended).

In any case – and whatever size model you start with – remember these words of caution: always check the generated result before using it as gospel truth. There will be times when an AI creates bizarre or problematic results; always trust, but verify! So go ahead, experimenters! Try building different sizes of generative AI models, why not? Experiment with training data sets. See how much creativity lies beneath its layers and layers of hyperparameters.

Just remember: “Size doesn’t matter… Look at me… Judge me by my size, won’t you?” -Yoda