Data Science Architect –Defining a ‘new’ role

The role of an architect in most areas of technology is well defined. For example, a Java architect will require expertise which might vary by project, domains and organizations, but the essential skills are technical, with a deep understanding of systems design and software architecture, and some project managerial experience.  

Even in data focused fields closely related to data science, like data engineering, the architect has a similar role across organizations and domains. The technology stack might be different and the creative ways of structuring projects, defining data relationships and ensuring data quality and privacy will be unique in every project, but with a similar central underlying theme.  

On the other hand, except knowing that a data science architect (DSA) is not a data architect, in data science the role of an architect is vague. Data science itself does not lie within a single academic discipline, unlike data engineering or Java which belong squarely to the world of computer science. This multidisciplinary flavor of the subject entails professionals in the area to have a multidisciplinary perspective, which increases when the usual managerial and creative demands of architecture kicks in.  

While multi-disciplinarity makes any job difficult, in the coming years as AI maturity, increase in data driven decision-making and the tangible relevance of analytical methods become obvious to traditional industries, the job of the DSA, which is rare today, becomes indispensable. Someone who can talk to the domain experts and the technology teams with equal ease, who can detect problems that can potentially be solved with high ROI and can plan out the entire process in detail and manage it across silos, can create enormous value in any industry.  

In our attempt to standardize this role we analyzed hundreds of data science leader and manager roles to extract the key elements, since data science architect is not even a common role in the leading job portals of the world. So, what are the key skills that we find necessary for data science architects? Broadly, we can categorize them into 5 buckets. This is over and above their skills as a data scientist.  

Problem Detection and Translation: The ability to detect problem areas which are ripe for data driven transformation using tools from the consulting world. It also involves business problems into data problems, and once the data driven solution is provided, translate back data solutions to business solutions

Domain Knowledge (depth): Every area where data science is used has some unique features and challenges. If the architect does not have perspective or experience in the domain, they will not be able to look for the solutions in the right place.  

Research and Experimental Mindset: Even today, there is no ‘canon’ which can tell a data scientist what they should be trying to solve most problems. While in certain domains it is somewhat set, even in those researching for the latest and greatest will be an essential skill.  

What to try? How to experiment? What to control for and why? Which models might make sense? What models from other disciplines can help understand and predict better? These are all a part of the experimental learning process for data scientists which needs to come from the experience of a DSA.  

Project Architecting (breadth): There are many moving parts in a data science project, each one of which contributes to its success. The DSA must be (while not an expert but) well conversant in each one of those. To begin with the DSA should be able to frame the problem and solution plan in simple frameworks like CRISP-DM.  

The DSA should understand enough about data engineering and big data to understand the data pipeline, realize the scalability potential of solutions, and its related challenges. There should be enough understanding about cloud services and costs to clearly calculate potential expenses and budget for a project. Lastly, the DSA should also know enough about the business to realize the Return on Investment that the project can provide.

Management and technical skills: Like every architect, the DSA also needs to have significant project, time management and personnel management skills to make the project a success.

Additionally, the DSA should have exceptional communication and negotiation skills because data will always sit in silos and internal/external clients will nearly always get frustrated with months of data engineering and cleaning, where they will see little of the magic that they expect from data science or AI

Technical expertise would include the skills of a senior or lead data scientist, but possibly diluted in some manner to ensure that they do not spend their entire time writing code or developing algorithms but architecting the solutions.  

The DSA will become an integral part of every organization in the next decade and many parts of the job will get more standardized. Till then, this might provide a template for the first generation with the Data Science Architect job title.

Dipyaman Sanyal

---

Dipyaman Sanyal is Founder and CEO of dōnō consulting.