The future of data science and what it means for you

One of the most obvious consequences of the commodification of data science is that there will be less demand for traditional data scientists as we know them. As more tasks become automated, the need for highly specialised professionals to perform those tasks will decrease. This is not to say that data scientists will become obsolete, but rather that the role will likely shift in other directions.

Opportunities for businesses

At the same time, the commodification of data science opens up new possibilities for smaller companies that may not have had the resources to invest in sophisticated data analytics in the past. With tools like AutoML, it's now possible for even relatively small businesses to build predictive models that can help them make better decisions and improve their bottom line.

So, what should businesses do in light of these developments? The answer is simple: embrace the tools that are available, but make sure you still have people who know what they are doing. While AutoML, ChatGPT and other automated tools can make it easier to build models, it's still important to have someone on your team who understands the underlying principles of statistics and machine learning. Additionally, it's important to be careful about building on top of tools that might change or disappear at any moment. In other words, use these tools to help development where possible, but don't rely on them as the backbone of your data science strategy.

But what about us data scientists?

The first step in adapting to the changes in the data science world is to recognise that indeed the demand for the type of traditional data scientist will continue to decrease. Naturally this process will take a while, but now is the time to start thinking about your future direction. I would say, there are three main directions to consider as a data scientist.

1. Become more business-savvy:

One way to stay relevant as a data scientist is to become more business-oriented. This means developing a deep understanding of the business problems that can be solved using data, and being able to communicate those solutions to stakeholders in a clear and compelling way. Business-savvy data scientists may also be responsible for helping to identify new data-driven opportunities for their organisation, and for designing and guiding the implementation of these solutions.

For example, a data scientist working for an e-commerce company might use their business expertise to analyse customer purchase patterns and develop recommendations for how the company can improve its product offerings. They might also help the company build a data-driven marketing campaign, using customer data to personalise promotions and increase engagement.

2. Move towards engineering and/or architecture:

Another path for data scientists is to move towards engineering and/or architecture. This means developing a deep understanding of the technical infrastructure required to build and deploy data-driven solutions. You’ll also need to be able to design and implement that infrastructure in a way that is scalable and efficient.

For example, a data scientist working for a bank might be responsible for designing and implementing a machine learning pipeline that can process large amounts of transaction data and generate insights in real-time. They might also work closely with software engineers and data engineers to ensure that the pipeline is robust and scala

3. Deep specialization

Finally, data scientists can choose to specialise deeply in a particular area of data science. This might mean developing expertise in a particular statistical technique or machine learning algorithm, or focusing on a specific domain such as healthcare or finance.

For example, a data scientist working for a pharmaceutical company might specialise in developing predictive models for drug discovery. They might spend their time researching and experimenting with different machine learning algorithms, or working closely with domain experts to understand the nuances of the drug discovery process.

One thing you should not do is despair. Despite the changes that are happening in the field, the future of data science is still exciting. As tools become more accessible and data becomes more ubiquitous, the possibilities for what can be achieved with data science are virtually limitless. The challenge for professionals is to stay ahead of the curve and remain adaptable in the face of these changes.

Why hire a consultant in this changing reality

Lastly, why would anyone still hire a data science consultant if data science is now so easy to do? After all, it is in many cases possible to train an internal employee to do machine learning projects relatively quickly using modern tooling. But there are a few reasons why it still makes sense to hire a data science consultant.

First, most consultants bring more than just data science or machine learning skills. For example, many are proficient in machine learning engineering as well, and therefore bring their own work to production and work very effectively with other engineers in the organisation. Another skill that many consultants bring is a strong business sense, especially in the case where they have experience working with a number of organisations and teams. Defining the problem in the right way to bring business value and integrating that value seamlessly into the existing business processes is a valuable skill. And one that is unlikely to be automated any time soon.

Second, an experienced consultant is capable of bringing an organisation to the next level by structuring teams and processes around projects. Additionally, a consultant knows how to implement cohesive strategies that can be replicated within the organisation after the consultant’s assignment is completed.

By leveraging their expertise in both the technical and business aspects of data science, consultants can remain relevant and valuable in a world where automated tools are becoming more prevalent. It’s also important to remember that by hiring a consultant from a company, you not only get that person, but the combined skills and knowledge of all employees of that company. In a culture like ours, we share experience and discuss problems and solutions constantly. Even if a challenge in your organization is not completely familiar to the person you hired, there is always a colleague who can lend their expertise to virtually any problem.

Exciting opportunities ahead

All in all, there is no reason to despair whatever your situation. It’s always good to be aware of changes in such a rapidly evolving field. It’s this awareness that allows you to anticipate the impact of changes on the horizon, to face them well prepared. But ultimately I strongly believe the future of data science is just as exciting as ever, and offers many interesting opportunities no matter where you are currently in your journey.