A few months ago, I wrote a piece for Fast Company talking about the roots of the data science skills gap. I called it a hiring hot mess because the underlying problem is a lack of understanding about what skills companies need to be successful with data science and machine learning initiatives. The result is job opportunities that have every requirement you can think of, so nothing is left out.
That is created our current obsession with data science unicorns. However, there simply are not enough to meet the rising demand for data science talent. Bootcamps, certification programs, and education accelerators have sprung up to try and train our way out of the shortage. The results are inconsistent. Rock stars emerge from these programs but often, the students come out semi-prepared for their first job.
This problem will not be confined to the data science and machine learning boom. The Internet of Things, Machine Learning at the Edge, Cyber Security, Quantum Computing, and Advanced Robotics will each create their own unicorn obsession and talent shortage. Talent will continue to pool at top companies and be a barrier to entry for the vast majority.
Solving This Gap and Gaps to Come – Embracing the New Normal in Talent
The hiring process is driven by the reality that the talent has always just been there. Some skills are harder to find than others but not in a way that disrupts the business’s ability to staff departments. Companies plan/budget between 1 and 3 years out for their hiring needs. They post job openings quarterly and as people leave. Several qualified people apply, and they hire the best candidate.
The reality for data science hiring is that several qualified people do not apply. Most applicants are lacking in skills or experience. Those who are qualified are already employed.
If you compare data science job openings on Indeed to people in data science on LinkedIn, you will see that there is an open job for every 2 employed data scientists. There’s little incentive to hop companies because there is a wink and a nod agreement in tech that prevents a price war for talent. If you look at industries like finance that don’t have the same collusion, you see experienced data scientist compensation packages running upwards of $300K per year which is why that industry isn’t having trouble attracting or retaining top talent.
Won’t Everything Even Out Eventually?
Conventional wisdom argues that this demand will incentivize more people to enter the field and that will stabilize the situation. There are 2 forces which will interfere with this natural direction of the market. The first is how long it takes to train a data scientist. The second is how quickly the field is evolving.
There are a lot of recent high school graduates who have the talent and inclination to enter the field. It will be about 6 – 8 years before they are ready to apply for their first data science job.
There are fewer, but still a significant number, of recent Bachelor’s graduates with the inclination to enter the field. It will be about 2 – 4 years before they are ready to apply for their first data science job.
There are significantly fewer Master’s and PhD. holders who are inclined to enter the field. It will be 1 – 2 years before they are ready to apply for their first data science job. They have career alternatives that are equally lucrative and do not require more training or experience to start. That means the best pool for bringing talent to the field rapidly has the least incentive to join.
The field is also evolving at a rapid pace. The skills aspiring data scientists are being trained on today will be different than what is considered current in a year or 2, let alone 3 – 6 years from now. A data scientist taking a year off from computer vision or natural language understanding work means taking a month or 2 to get back up to speed with the latest developments if they return to the niche.
I used to be competent across a broad spread of industries and use cases when it came to data science and machine learning. Every year the pace of change is forcing me into deeper specialization across fewer industries and significantly fewer use cases. I used to keep up with 8 different programming languages’ data science tools. Now I have pared down to 4. Specialization is not bad at all, but I want to drive home the point that those of us in the field spend a significant amount of time staying current with our niche(s). Those coming into the field have a hill to climb before they are ready to contribute at a high level.
The long lead time to job readiness coupled with a continually moving goal post makes conditions right for a prolonged correction. Eventually conditions will normalize, and the talent gap will subside. However, as I mention at the beginning of the post, this is the first wave of talent gaps and not the last. Failing to resolve a broken pipeline of high-end talent will only result in repeating the mistakes of the past.
Connecting Supply and Future Demand
The assumption that talent will simply be there is no longer accurate. In the early 90’s, IGT was having troubling finding enough talent in Nevada to fill their software and hardware engineering needs. They partnered with the University of Nevada in Reno and Las Vegas to help shape the computer science curriculum. The effort was a success and the partnership continued into the early 2000’s. Around the same time Microsoft gave the University of Nevada, Reno a large grant to work on computer vision. The grant helped push education about computer vision, and by extension machine learning, into all levels of computer science classes. That is how I got interested in the field so, thanks Microsoft!
With a few updates, these same partnerships can solve our current and future skill gaps. First, we need to align businesses’ technology roadmaps with college or boot camp curriculum in a forward looking way. We cannot keep teaching today’s skills to students who will not hit the workforce for a year or more. We need to be pulling the programming languages, platforms, and features off technology roadmaps and into the classrooms. That is a shift in thinking for academia because it means embracing up and coming technology rather than waiting for technology to establish itself before it hits the classroom.
Technology is not static, so classes need to release updates to students throughout their education with the latest developments. That means a semester class on Python for machine learning keeps sending quarterly updates that cover additions to the language as well as the latest libraries. These updates need to be heavily influenced by how/if businesses are leveraging these changes. Again, the partnership between businesses and educational institutions helps make the curriculum more relevant.
There also needs to be an exchange program between educational institutions and businesses. Teachers head to work in the business to get hands on experience while engineers and data scientists come into the classroom to bring the real-world in. This is one of the huge benefits of boot camps taught by working professionals; practical know how. However, the exchange also preserves the professional educator’s place in the classroom. Teaching is a skilled craft honed over years of experience.
Removing the Barrier Between Office and Classroom
The largest complaint on both sides of the data science hiring equation is experience. Businesses want candidates who can start contributing as soon as possible so they usually require experience. Aspiring data scientists are left wondering, ‘How do I get experience if no one will hire me?’
How we can allow anyone to graduate or be certified without a single year of work experience is something that has bothered me for almost 20 years. Businesses have been complaining for decades that students graduate unprepared to actually do the job in a number of fields. We need to integrate students into real world functional teams as part of the educational process.
I think it would be an excellent end of year evaluation versus test/project-based grading. Have students spend three months working and the work evaluation they receive is their grade. A failing evaluation means they need to repeat the curriculum.
Companies would know which students they want to hire, and the evaluation process is far more comprehensive than a job interview. New hires would be partially integrated into the team and way the business operates. They would be proven and ready to contribute right after graduation.
The cost of 50 evaluations, about 12.5 full time equivalents, is around $600K per year. That is not an insignificant investment. It returns quickly in tangible sourcing and onboarding costs. It also drops the amount of time positions are vacant to near 0. It adds predictability to the hiring cycle and certainty to candidate fit/capability.
If I walked into a client’s office today and said I can provide these benefits for $600K/year, it would still be a tough road to getting consensus and buy in. Getting any business to invest in staffing related initiatives is an uphill battle unless the problem is hurting the business. I think with data science staffing we have a pain point that is causing enough business disruptions to support an investment in a solution.
A Lot of Partial Solutions but We Need A Longer-Term Fix
A lot of these types of partnerships have materialized over the last two years. Most of them are purpose built for data science and machine learning. That is not a bad thing for the field and its place in business. However, it does not solve the larger problem nor address any of the talent gaps to come.
What is needed is a closer relationship between business and academia, both colleges and alternative education platforms. We need to blur the lines between working and learning. Most students are learning to apply that education in a rapidly changing real world. Most workers will never stop learning.
The age of doing the same thing for an entire career is well behind us. However, our educational architecture has not evolved to meet the need for continuous, practical, forward looking learning. Forward looking education also needs to extend to reskilling workers who will soon be displaced by technology. We need to be reaching out to truck drivers and factory workers today who will have their livelihoods disrupted in the next 5 – 10 years. The process of educating them into sustainable careers needs to start now, not after the layoffs.
In 10 – 15 years we will need to start reskilling the software developers and data scientists we have just finished educating. This process will be perpetual, and we do not have a framework to handle it. A close tie between business and academia as well as a forward looking, practical curriculum will go a long way towards establishing that framework. Creating an obvious transition path between education and employment is another fundamental piece that is missing.
I wrapped up my first article by suggesting that the real skills gap was a lack of connection between data science skills and business needs. The root cause for that disconnect is the lack of connection between academia and business.