Data Engineering Lead at Paystack

A lead data engineer must be capable of operating across a diverse tech stack. They are expected to be adaptable and unafraid of the unknown. The role calls for someone with a good background in software development, data analytics and distributed systems. Furthermore, the lead needs to provide their team with a good working environment and be able to manage stakeholder expectations about their team’s abilities.

A lead data engineer’s responsibilities may include:

• Architectural design of a data platform
• Data pipeline maintenance/testing
• Administration of a large cloud data warehouse
• Creation and maintenance of infrastructure-as-code components
• Monitoring system performance and addressing faults in production systems with an on-call rotation
• Development of in-house software
• Regular check-ins with their direct reports
• Development of team rituals and administration

Requirements and skills

Technical skills

• A software engineering background
• Experience with test driven development
• Fluency in a modern programming language, including but not limited to: Java, Scala, Python, C/C++ and Golang
• An understanding of data modelling
• Experience in running ETL pipelines
• Expertise in data analysis
• Knowledge of BI tools
• Event streaming/message broker systems
• Change data capture
• Experience with kubernetes
• Familiarity with a cloud environment (AWS, Azure, GCP)
• Familiarity with MongoDB, Postgresql and mysql

Tools

• Kubernetes
• Kafka Connect
• Debezium
• Prometheus
• AWS Redshift
• dbt
• S3
• CI/CD pipelines
• Workflow orchestration tools (Airflow and Dagster specifically)

Soft skills

Communication skills

• Be able to communicate with different stakeholders (data analysts, machine learning engineers, CTOs, and developers)
• Be able to work with other teams or business units to gather requirements and define the scope of a project
• They must understand the underlying business problems that they are trying to address and articulate how their solution can help

Collaboration

• Be able to work closely with team mates in a remote setting
• Be able to deal with difficult problems and own their own mistakes