A lead data engineer must be capable of operating across a diverse tech stack. They are expected to be adaptable and unafraid of the unknown. The role calls for someone with a good background in software development, data analytics and distributed systems. Furthermore, the lead needs to provide their team with a good working environment and be able to manage stakeholder expectations about their team’s abilities.
A lead data engineer’s responsibilities may include:
- Architectural design of a data platform
- Data pipeline maintenance/testing
- Administration of a large cloud data warehouse
- Creation and maintenance of infrastructure-as-code components
- Monitoring system performance and addressing faults in production systems with an on-call rotation
- Development of in-house software
- Regular check-ins with their direct reports
- Development of team rituals and administration
Requirements and skills
Technical skills
- A software engineering background
- Experience with test driven development
- Fluency in a modern programming language, including but not limited to: Java, Scala, Python, C/C++ and Golang
- An understanding of data modelling
- Experience in running ETL pipelines
- Expertise in data analysis
- Knowledge of BI tools
- Event streaming/message broker systems
- Change data capture
- Experience with kubernetes
- Familiarity with a cloud environment (AWS, Azure, GCP)
- Familiarity with MongoDB, Postgresql and mysql
Tools
- Kubernetes
- Kafka Connect
- Debezium
- Prometheus
- AWS Redshift
- dbt
- S3
- CI/CD pipelines
- Workflow orchestration tools (Airflow and Dagster specifically)
Soft skills
Communication skills
- Be able to communicate with different stakeholders (data analysts, machine learning engineers, CTOs, and developers)
- Be able to work with other teams or business units to gather requirements and define the scope of a project
- They must understand the underlying business problems that they are trying to address and articulate how their solution can help
Collaboration
- Be able to work closely with team mates in a remote setting
- Be able to deal with difficult problems and own their own mistakes
How to Apply
Interested and qualified? Go to Paystack on boards.greenhouse.io to apply