Data Engineer Iii

Overview

• Design and implement advanced data solutions to support business operations and analytics.
• Lead technical initiatives and mentor junior team members in data engineering practices.
• Drive innovation in data pipeline development and optimization using cutting-edge technologies.
• Collaborate with stakeholders to align data solutions with business reporting requirements.
• Ensure data quality and reliability through rigorous testing and troubleshooting.
• Contribute to the modernization and improvement of the organization's data platform.
• Support continuous integration and deployment processes to streamline development workflows.
• Participate in schema evolution and indexing strategies to enhance database performance.
• Utilize Azure cloud services to implement scalable and secure data solutions.

Key Responsibilities & Duties

• Develop and maintain incremental data pipelines using SSIS, PySpark notebooks, and Python.
• Implement and optimize ETL/ELT processes for Azure SQL databases and data lake environments.
• Build meta-driven, scalable data ingestion frameworks supporting SCD Type 2 methodology.
• Write and optimize T-SQL stored procedures, views, and dynamic SQL for data transformations.
• Develop Python-based automation including Azure Functions, blob triggers, and API integrations.
• Maintain data quality, perform root cause analysis, and resolve data pipeline issues.
• Support CI/CD processes including code reviews, Git version control, and Azure DevOps workflows.
• Monitor pipeline performance and implement optimization strategies.
• Assist with data modeling, indexing strategies, and schema evolution.

Job Requirements

• Bachelor of Science (BS) degree in a relevant field.
• Minimum of 5 years of experience in data engineering or related roles.
• Proficiency with SQL Server, T-SQL, and database concepts (indexing, query optimization).
• Experience with ETL/ELT tools and concepts (SSIS or similar preferred).
• Understanding of data warehousing, dimensional modeling, and SCD methodologies.
• Familiarity with Azure cloud services (especially Azure SQL Database).
• Proficiency in Python programming for data engineering and automation.
• Knowledge of CI/CD practices and Azure DevOps workflows.
• Experience with modern data platforms, including lakehouse architectures and Delta Lake.