Job VC

Key Responsibilities

Design, build, and maintain scalable, reliable data pipelines (batch and streaming).
Develop and optimise data models, warehouses, and lakehouse architectures.
Build and automate CI/CD pipelines for data workflows and deployments.
Ensure data quality, integrity, lineage, and governance across environments.
Implement proactive monitoring and alerting for pipelines and data infrastructure (CloudWatch, Grafana).
Manage IaC for consistent, reliable data infrastructure.
Troubleshoot and optimise pipeline and query performance across environments.
Lead incident resolution and root cause analysis for data systems.
Collaborate with analysts, data scientists, and engineering teams, ensuring security, scalability, and operational excellence to understand data requirements and translate them into technical solutions
Document data architecture, lineage, and workflows

Must have

5+ years in data engineering, with solid DevOps/platform engineering exposure.
Strong SQL and data modelling skills (dimensional, normalised, lakehouse).
Proficient in Python; working knowledge of Scala, Java, or Go.
Hands-on experience with data pipeline/orchestration tools (Airflow, DBT).
Experience with distributed processing frameworks (Spark, Flink, or similar).
Advanced AWS expertise (Glue, EMR, Redshift, Athena, S3, Lambda, Kinesis).
CI/CD tools (GitHub Actions, GitLab CI, Jenkins, CircleCI).
Infrastructure as Code (CDK, CloudFormation).
Experience with streaming/event-driven architecture (Kafka, Kinesis, Apache Flink, or Pub/Sub).
Observability tools: Grafana, CloudWatch.
Expertise with data warehouse platforms (Redshift, Snowflake).
Desirable

Experience with containerisation and orchestration (Docker, ECS, Kubernetes).
Understanding of data governance, cataloguing, and security/compliance (GDPR, lineage tools).
Experience with GitHub Enterprise, Atlassian APIs, and Slack bots.
Understanding of cost optimisation strategies in multi-cloud and data warehouse environments.