Job Title: Azure Data Engineer / Azure Cloud Data Engineer
Overview: We are looking for a talented Azure Data Engineer to join our dynamic team. As an Azure Data Engineer, you will play a crucial role in the assessment, planning, and seamless migration of on-premises SQL Server environments to Azure SQL. This position requires expertise in database administration, Azure services, and proficiency in C# programming for Azure Functions development.
Responsibilities:
Assessment and Planning:
· Evaluate existing on-site SQL Server environment.
· Collaborate with stakeholders to gather requirements for Azure SQL migration.
· Develop a comprehensive migration plan, considering downtime, data transfer, and compatibility.
Database Migration:
· Utilize Azure Database Migration Service (DMS) or relevant tools for seamless migration to Azure SQL.
· Ensure data integrity and consistency during the migration process.
· Minimize downtime and disruptions to ongoing operations.
Azure SQL Configuration:
· Configure Azure SQL databases, servers, and related services based on requirements.
· Optimize Azure SQL performance and scalability.
· Implement security measures, including access controls and encryption.
Replication of Processes:
· Replicate existing on-premises SQL Server processes and jobs in Azure SQL.
· Ensure scheduled tasks, backups, and maintenance plans are transferred and configured appropriately.
Data Ingestion from 3rd Party Vendors:
· Understand APIs provided by various 3rd party vendors.
· Design and implement robust data ingestion processes tailored to each vendor's API.
· Develop efficient workflows for the extraction, transformation, and loading (ETL) of external data into Azure SQL databases.
· Implement SQL replication when necessary for data synchronization.
· Ensure the security and integrity of ingested data from external sources.
Azure Services Integration:
· Utilize Azure Data Factory, Logic Apps, or other relevant services to seamlessly integrate external data sources into the Azure SQL environment.
· Leverage Azure Data Bricks or similar tools for large-scale data processing and transformation.
· Design and implement Azure Functions for specific tasks or processes.
· Integrate Azure Functions into workflows for data processing and transformation.
Performance Monitoring and Optimization:
· Implement monitoring solutions for Azure SQL to track performance metrics and ensure the completion of data migration functions/jobs. Enable the ability to restart these functions/jobs in case of errors and configure SMS/Email notifications for severe issues.
· Identify and address performance bottlenecks; optimize queries for efficiency.
Troubleshooting and Support:
· Provide support during and after migration to address any technical issues.
· Collaborate with Azure support services when needed.
Qualifications:
· Bachelor’s degree in computer science, Information Technology, or a related field.
· 3-4 years of experience as an Azure Data Engineer or in a similar role.
· Strong background in database administration, particularly with SQL Server and Azure SQL.
· Experience with Azure services, including Azure Functions and Azure Queues.
· Proficiency in C# programming language, with a focus on developing Azure Functions for specific tasks or processes.
· Proven expertise in data ingestion from 3rd party vendors using REST APIs and SQL replication.
· Knowledge of data engineering tools, ETL processes, and data warehousing.
· Azure certifications (e.g., Microsoft Certified: Azure Data Engineer Associate) are a plus.
· Excellent communication and collaboration skills.
We are an equal opportunity employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, national origin, sex, sexual orientation, age, disability, gender identity, marital or veteran status, or any other protected class.
