Senior Data Engineer ? Dublin (Hybrid)
Contract Role | 3 Days Onsite
We are seeking an experiencedSenior Data Engineerto join our high-performing data team in Dublin. This is an exciting opportunity for a contractor who thrives in complex, large-scale environments and enjoys working with modern data platforms. In this role, you will design and optimize critical data pipelines across the Hadoop and Spark ecosystem, work extensively with Snowflake and Databricks, and contribute to the transformation of our big-data environment into a more scalable, cloud-ready architecture.
Key Responsibilities
Data Engineering & Pipeline Development
- Build, enhance, and maintain large-scale ETL/ELT pipelines usingHadoop ecosystem toolsincludingHDFS, Hive, Impala, andOozie/Airflow.
- Develop distributed data processing solutions withPySpark, Spark SQL, Scala, or Pythonto support complex data transformations.
- Implement scalable and secure data ingestion frameworks to support both batch and streaming workloads.
- Work hands-on withSnowflaketo design performant data models, optimize queries, and establish solid data governance practices.
- Collaborate on the migration and modernization of current big-data workloads to cloud-native platforms andDatabricks.
Platform Architecture & Optimization
- Tune Hadoop, Spark, and Snowflake systems for performance, storage efficiency, and reliability.
- Apply best practices in data modelling, partitioning strategies, and job orchestration for large datasets.
- Integrate metadata management, lineage tracking, and governance standards across the platform.
Data Quality & Test Automation
- Build automated validation frameworks to ensure accuracy, completeness, and reliability of data pipelines.
- Develop unit, integration, and end-to-end testing for ETL workflows using Python, Spark, and dbt testing where applicable.
- Implement reconciliation checks, regression suites, schema validations, and anomaly detection tools.
Cloud & Databricks Integration
- Work with cloud services such asAWS S3, Glue, Lambda, Athena, EMR,Redshift, or equivalent platforms.
- Support the adoption and expansion of theDatabricks Data & AI Platform, contributing to Delta Lake implementation, job scheduling, and ML-ready pipelines.
- Participate in cloud modernization efforts and architectural redesigns focused on performance and cost efficiency.
Collaboration & Leadership
- Partner closely with product, engineering, and data science teams to ensure quality and performance across the data lifecycle.
- Participate in agile ceremonies and improve sprint planning with data-driven insights.
- Mentor junior engineers, promote best practices, and help shape the long-term technical roadmap.
- Contribute to incident response, troubleshooting, and root-cause analysis for production issues.
Required Experience & Skills
Technical Expertise
- 7+ years of experienceas a Data Engineer working with distributed data systems.
- 4+ years of deep Snowflake experience, including performance tuning, SQL optimization, and data modelling.
- Strong hands-on experience with theHadoop ecosystem:
- HDFS, Hive, Impala, Spark (PySpark preferred)
- Oozie, Airflow, or similar orchestration tools
- Proven expertise withPySpark, Spark SQL, and large-scale data processing patterns.
- Experience withDatabricksand Delta Lake (or equivalent big-data platforms).
- Strong programming background inPython, Scala, or Java.
- Experience with cloud services (AWS preferred):S3, Glue, EMR, Redshift, Lambda, Athena, etc.
