Data science has come a long way, from the days when projects resembled building monumental pyramids to the current era, where large language models (LLMs) have transformed the landscape, allowing us to manage icebergs instead of erecting titanic structures. In this article, we will explore this situation and how LLMs are redefining the traditional approach to artificial intelligence (AI) and data science projects.
Data science as pyramid building
Traditionally, data science projects resemble pyramid building. In these projects, the base of the pyramid is crucial: it involves collecting, extracting, cleaning and transforming data – tasks that require considerable effort. This is where the human factor comes into play: teams of data engineers, programmers and data scientists work tirelessly to build a solid foundation on which an AI model can be erected.
Once this foundation is complete, you can begin building the model, which is designed to solve a specific problem, such as classifying tweets or predicting energy demand for a power company. This approach is laborious and, in many cases, requires massive volumes of data to train effective models. Each layer of the pyramid represents a phase of the project, from data manipulation to modeling and implementation. Without a solid foundation, the pyramid collapses.
LLMs: From the pyramid to the iceberg
The advent of LLMs has radically changed this trend. Instead of building a pyramid from scratch, we are now faced with a floating iceberg, where most of the structure is hidden underwater. The LLM represents this massive, pre-trained knowledge that already exists and is available from minute one. This means that instead of starting from the ground up, data science teams can leverage this accumulated knowledge and focus on the visible part of the iceberg: model customization and adaptation to specific tasks.
With LLMs, the focus has shifted from “building” to “managing.” The key here is the ability to use techniques such as prompting, which allow the LLM to be guided to behave in the desired way with significantly less effort. Rather than needing large volumes of data to train a model from scratch, in many cases a limited set of examples and good prompt design are enough to yield outstanding results. This not only reduces the time and resources required, but also democratizes the access to advanced AI, allowing more organizations to take advantage of its benefits.
The impact of LLMs on data science projects
The shift from building pyramids to managing icebergs has profound implications for how data science projects are conducted. Here are some of the most notable benefits:
- Cost and time reduction: By starting from a pre-trained model, the time and resources required to develop AI solutions are drastically reduced.
- Democratizing access to AI: Organizations that previously lacked the resources to build pyramids can now manage icebergs, accessing advanced AI capabilities without the need for large infrastructures or specialized teams.
- Flexibility and adaptability: LLMs allow for greater flexibility as they can be quickly adapted to new tasks with minimal human intervention, allowing companies to respond faster to market changes.
- Focus on innovation: By reducing the workload of data preparation and building models from scratch, teams can focus more on innovation and how to strategically apply AI to their businesses.
Conclusion
The pyramids and icebergs metaphor perfectly captures the transition in data science driven by LLMs. While pyramid building remains relevant for certain types of projects where precise, tailored control from the ground up is required, iceberg management is quickly becoming the norm for many applications, thanks to the ability of LLMs to tap into vast volumes of pre-existing knowledge.
As we look to the future, we are likely to see a combination of both approaches, where data scientists build pyramids when necessary, but also know how to navigate and manage icebergs when circumstances allow. This duality is what will make data science continue to evolve and provide value in an increasingly AI-driven world.
And remember, if you need help building pyramids or managing icebergs, at catedra.ai we will be happy to help you. Don’t hesitate to contact us.
