Without capable leaders, machine learning teams fail to thrive. Managing is pushing papers from one side of the desk to another. Leadership is making the team more capable. There is an accountability for team productivity and project success. Leaders are responsible for building relationships both in the team and with the rest of the business.
The highest performing teams do not always have top talent. They do have the best leaders. The concepts of mentoring and development are central to great leadership. Communications are critical. Leaders build a team identity and standards. They take a collection of individuals and build a cohesive unit that is more productive together.
In this post, I am going to cover each one of those points. Leaders in the machine learning field face completely different challenges than most leaders, even in other technical areas. I am going to start by explaining why canned leadership styles and traits do not transfer well. I will then cover what has worked for me over the last 10 years.
The Challenges of Leading A Data Science Team – What Does not Work
I have friends who ask me to come work for them. I respond, “I like you too much to do that to you.” Data scientists are a challenge to lead because we deny leaders their traditional sources of authority. Add to that the challenges of leading high performers and the complexity of the work itself. Motivational posters and pop psychology are not effective.
Data scientists resist leadership. Some have been consultants or worked in small teams. They have had a high level of autonomy and little to no oversight. Others have only worked with ineffective leaders. They see the role as an impediment to progress. Some have worked with leaders who were controlling, belittling, and ridiculed ideas that were not theirs.
The titles of manager, director, etc. are not a source of authority. We cannot, “Because I said so” or “I am in charge here” our way into success. I have seen leaders try to fire they way to authority. This approach leads to the best data scientists leaving and being replaced by mediocre talent who just nod their heads without question.
Competence is a common source of authority in technical teams. Often the best engineer is promoted to a leadership position because the team follows that person organically. This is also true in data science teams. Technical leadership is a small part of the overall role.
Even competence eventually fails because there is always someone more knowledgeable. Machine learning is too complex for one person to be the most competent across every project. One of the cornerstones for leading a data science team is the leader must be able to relinquish technical leadership without losing their source of authority. We lead experts while rarely being the most knowledgeable.
Two Sources of Authority That Work
The first is alignment of the company’s goals with the team’s’ goals. It is basic game theory. Create an equilibrium where the activities of the data science team lead to success for both the business and the data scientists. Leaders need to take the time to understand each data scientist’s personal goals and motivators then build a structure that rewards the data scientist for activities that create business value. Alignment is a powerful source of authority because the data scientist is internally motivated by those layered goals.
There is a second piece to alignment, connecting team goals with individual goals. Data science should be a collaboration between all team members to avoid ‘Not My Project’ syndrome. It is easy for data scientists to focus on their projects to the detriment of others’. Putting in place a system that rewards everyone for team success creates an incentive to jump in as needed. That framework creates accountability to each other instead of focusing accountability on the leader.
This is the setup for the second source of authority, trust. Aligned goals build the framework for the team to be a part of the larger company and for each data scientist to be part of the team. That allows each person to trust the team and larger business when they see both contributing to aligned goals. However, when they perceive, right or wrong, that a person or group is not fully supporting those goals, they will be just as committed to calling it out and insisting on a resolution.
I have learned that trust is a two-way street. A leader can only ask the team to follow them if that leader is willing to follow the team. It sounds like a motivational poster, but it is a truism for leading data science teams. Data scientists expect to be listened to and have their thoughts acted on. That means a data science leader is often in the position to manage and translate up to senior leadership.
A data science leader needs to be capable of pushing back on deadlines, projects, a lack of funding or staffing, and any other scenario what the team is being told to do work that doesn’t line up with their understanding of the goals. That disconnect requires a leader who can address the issue at whatever level is necessary.
The leader needs data science domain expertise to understand the details of the disconnect. They also need business acumen, so the leader can translate the disconnect into language decision makers understand. The team needs to trust the leader is not only willing to take their concerns up but capable of driving a sensible resolution.
In other cases, the leader uses those same skills to translate business cases to the data science team well enough that they build models to meet the need. The team must trust the leader as a conduit between them and external teams, leadership, and sometime users. That is an important piece of the data science leader’s value proposition and goes a long way towards the team seeing the leader as an enabler rather than an extra, unnecessary step in the chain.
Trust and alignment replace technical competence with leadership competence. Once the team understands the need for and the benefits of leadership, they are more willing to accept a leader. Acceptance is the end goal of alignment and trust-based leadership. While this is basic leadership theory, it is instructive to understand why other fundamentals of leadership theory fail while this approach thrives.
Mentoring and Growing a Team
Leaders are responsible for making the team more capable. Hiring and mentoring are their best tools to achieve that goal. What mentoring is needed? Who should the team hire? Does the team really need new talent?
People are capable of growth throughout their careers. Most are not encouraged to, especially once they reach a senior level. It is common sense that the machine learning field requires continuous education. Mentoring provides the opportunity, structure, and direction for that growth.
Career path and capabilities are the two main professional growth directions. Data scientists get bored and leave for a greater challenge. Growth retains talented people.
Most data scientists want to improve their value to the business and their ability to produce something tangible. A leader starts mentoring by showing someone their talent. It is hard to see your own talents. That honest self-assessment usually needs a mentor to act as a mirror.
Looking 6 to 18 months ahead, what capabilities does the business need to complete the next round of projects? That is how a leader provides direction. Present what the business needs and let them choose what interests them most. Focus on boosting their strengths. A lot of data scientists are chasing unicorn status. There is more benefit improving a skill from experienced to expert than from basic to mediocre.
Finally, create a realistic plan. What will it take to advance? How long? Is this something that another team member can teach? Is this an independent study project or learning path? Will this require an online or more traditional education plan? Again, opportunity, structure, and direction.
However, the company can only provide so much opportunity. A data scientist’s interests and capability growth plan may not line up with the business’s needs. A leader in often in a position where being a good mentor seems to lead to a talented data scientist leaving the company.
How does a leader mentor and retain at the same time? You have to get creative. What has worked for me in the past is fostering those interests as part of their role and part of the personal brand. Their role in the company can become more public facing. Publishing research and speaking at conferences are 2 solutions. The business gets exposure through the data scientist’s work and the data scientist starts to build their own reputation in the field.
Leaders are also mentoring their replacements and other mentors. Learn, apply, teach is the progression of capability growth. The long-term objective of mentoring is to teach them to mentor others. Senior team members need to be taught to mentor. Leaders need to find their replacements and teach someone else to do their job. You cannot get promoted if you are the only person capable of leading the team.
Hiring New Data Scientists
Once a leader understands their ability to teach and create a growth path for the team, it completely changes the rules of hiring. You do not have to hire data scientists with a massive skills list. The team is not focused on the perfect candidate.
A growing, cohesive team knows they can teach a lot of what a new hire needs to be a contributor. There is a greater confidence in their ability to bring a new hire up to speed. This mindset opens the door to candidates who will be long term contributors. The team is looking for quick learners, ambition, strong conceptual foundations, and communication skills.
As a leader, you also need to build a structured hiring process. Leaders network both inside and outside of the business. Internal promotions are the best pipeline for filling new roles. Building relationships with potential hires outside of the company is far more effective than trying to screen someone during an interview.
Leaders must create a repeatable, consistent interview and selection process. I hold a quick candidate prescreen with the team. We look over a shortlist of candidates’ profiles and reduce the list to no more than three interviews.
Each phase of the interview process has a list of questions. Follow up questions are allowed and can be less structured. I setup evaluation points and interviewers give candidate a score on each point. The highest overall score gets offered the job.
The process is structured, repeatable, and fair. Each person in the interview gets an equal voice.
A Leader Needs a Growth Plan and Mentor
I am wrapping up with something leaders can overlook. You spend so much time focusing on growing the team, you can neglect the need for your own growth. Leaders need mentors. Leaders need a career path and personal growth plan.
Leadership grows from team to teams to organization to business unit to business. This post covers leadership at the data science team level. That is all you may want to do. Spending a large part of your career leading and growing data science teams can be very rewarding. You will still need a mentor because there is always room for improving your leadership style and effectiveness.
Leadership’s career track can lead to starting your own business. A mentor can help you there as well. In many cases, a good mentor will mentor you past them. They will set you up to become better than they are.