Job VC
Senior Data /Databricks Engineer (contract/remote)
Technologies
Description
The Role
As a Core Data Engineer, you will play a pivotal role in designing, building, and optimising data pipelines, architectures, and workflows for a diverse client base. You’ll work within dynamic, multi-disciplinary project teams to implement scalable data solutions while adhering to best practices and emerging trends in data engineering.
Key Responsibilities:
Data Pipeline Development: Design and implement robust, scalable, and efficient data pipelines to collect, transform, and integrate data from various sources.
Data Architecture: Develop and optimize data architectures, including data warehouses, data lakes, and other storage solutions.
ETL/ELT Processes: Create and maintain reliable ETL/ELT workflows that ensure data quality and accessibility.
Collaboration: Work closely with data scientists, analysts, and client stakeholders to understand requirements and deliver impactful solutions.
Performance Optimisation: Monitor and enhance the performance of data systems to ensure minimal downtime and fast query responses.
Automation and Tools: Identify opportunities for automation and recommend tools to improve data engineering workflows.
Documentation: Maintain detailed technical documentation for all solutions and processes.
Technical Skills:
Programming: Proficiency in Python, Java, Scala, or similar languages for data processing.
Big Data Technologies: Hands-on experience with big data tools e.g., Databricks, Apache Spark, Hadoop.
Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks.
Database Management: Expertise in relational databases (e.g. postgres, sql server)
Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend.
Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design.
Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines for deployment.
Experience
2+ years of experience in data engineering or a similar role.
Proven track record of delivering data solutions in a consulting or client-facing environment is a plus.
Experience with Agile or Scrum methodologies is beneficial.
Personal Qualities
Strong problem-solving and analytical thinking abilities.
Excellent communication skills to explain complex technical concepts to non-technical stakeholders.
A team-oriented mindset with a focus on collaboration and adaptability
As a Core Data Engineer, you will play a pivotal role in designing, building, and optimising data pipelines, architectures, and workflows for a diverse client base. You’ll work within dynamic, multi-disciplinary project teams to implement scalable data solutions while adhering to best practices and emerging trends in data engineering.
Key Responsibilities:
Data Pipeline Development: Design and implement robust, scalable, and efficient data pipelines to collect, transform, and integrate data from various sources.
Data Architecture: Develop and optimize data architectures, including data warehouses, data lakes, and other storage solutions.
ETL/ELT Processes: Create and maintain reliable ETL/ELT workflows that ensure data quality and accessibility.
Collaboration: Work closely with data scientists, analysts, and client stakeholders to understand requirements and deliver impactful solutions.
Performance Optimisation: Monitor and enhance the performance of data systems to ensure minimal downtime and fast query responses.
Automation and Tools: Identify opportunities for automation and recommend tools to improve data engineering workflows.
Documentation: Maintain detailed technical documentation for all solutions and processes.
Technical Skills:
Programming: Proficiency in Python, Java, Scala, or similar languages for data processing.
Big Data Technologies: Hands-on experience with big data tools e.g., Databricks, Apache Spark, Hadoop.
Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks.
Database Management: Expertise in relational databases (e.g. postgres, sql server)
Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend.
Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design.
Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines for deployment.
Experience
2+ years of experience in data engineering or a similar role.
Proven track record of delivering data solutions in a consulting or client-facing environment is a plus.
Experience with Agile or Scrum methodologies is beneficial.
Personal Qualities
Strong problem-solving and analytical thinking abilities.
Excellent communication skills to explain complex technical concepts to non-technical stakeholders.
A team-oriented mindset with a focus on collaboration and adaptability