Position: Software Engineer
Department: Data Engineering & Analytics
Employment Type: Full-time
Role Overview
We are seeking a highly skilled and motivated Software Engineer with strong expertise in data engineering and analytics. The ideal candidate will have hands-on experience with modern data lake architectures on Azure or AWS platforms, with a particular focus on the Azure stack. This role will involve designing, developing, and optimizing data pipelines, enabling advanced analytics, and leveraging scalable big data solutions.
Key Responsibilities
- Design, build, and maintain robust, scalable, and efficient data pipelines for ETL/ELT processes.
- Develop and manage data lake architectures using Azure or AWS, ensuring high availability, performance, and security.
- Implement and optimize data solutions using Databricks, Azure Data Factory, Delta Lake, and other modern tools.
- Collaborate with analytics and data science teams to enable access to curated datasets and insights.
- Implement and manage NoSQL databases and big data processing frameworks like Apache Spark.
- Develop and document best practices for ETL patterns, data modeling, and governance.
- Monitor, troubleshoot, and improve the performance of data workflows and systems.
- Stay updated on emerging data engineering technologies and contribute to the adoption of innovative solutions.
Required Skills & Experience
- Proven experience in data engineering and analytics roles, with a focus on cloud-based platforms.
- Expertise in working with Azure or AWS ecosystems. Strong preference for Azure, including:
- Azure Data Factory
- Azure Databricks
- Delta Lake
- Azure Synapse Analytics (optional but beneficial)
- Proficiency in big data technologies like Apache Spark and Delta Lake.
- Strong experience with NoSQL databases (e.g., CosmosDB, MongoDB, DynamoDB).
- Solid understanding of ETL/ELT patterns and hands-on implementation experience.
- Proficiency in programming languages such as Python, Scala, or SQL for data manipulation and pipeline development.
- Familiarity with modern data governance and security practices.
- Excellent problem-solving skills and attention to detail.
Preferred Qualifications
- Experience with Azure Data Explorer or Stream Analytics.
- Familiarity with DevOps practices, including CI/CD pipelines for data workflows.
- Knowledge of data modeling and data warehouse design principles.
- Hands-on experience with real-time data processing and event-driven architectures.
