Advertisement
Job Description:
Responsibilities
Be the owner for data engineering solutions, ensure the continuity of data processes and the associated batch jobs.
Identify, design and implement/coordinate implementation of scalable processes and infrastructure to have good governance of automated data processes.
Manage the end-to-end data solutions of our customers: from raw data analysis to data flow and predictive framework configurations.
Should be able to work on a problem independently and prepare client ready deliverable with minimal supervision. Elicit, analyze, and validate customer data from ingestion to production
Monitor all data update processes and outputs to ensure predictive quality
Communicate with customers to discuss any issues with received data and help them identify and fix data issues
Solve day-to-day Data problems and customer challenges
Own the automation, deployment and operation of data pipelines on MS Azure
Build tools and mechanism to monitor and optimize different parts of the systems
Build custom integrations between cloud-based systems using APIs
Skills and qualifications
Required:
Expertise in ETL tools like Informatica/ADF
Familiarity with Linux/Unix scripting, advanced SQL proficiency and using various languages (Python, JavaScript, R, SnowSQL) to create sprocs, functions, DDL, and DML
Exposure to data modeling, creating conceptual and logical model, create DFD, ERD to understand relationship of data and possess good Database concepts.
Exposure to cloud onboard from legacy data sets.
Working experience on Azure Data platform, Apache Spark distributed processing system and Cloud computing
Demonstrates a knowledge of Azure cloud and networking architecture including various storage types: file storage, containers (example)ADLSG2, Azure SQL databases , Azure Data Factory, Azure Databricks and other cloud integration capabilities like Logic Apps, Function Apps, Serverless technology, VNet etc
Experience in creating data orchestration using ADF, optimizing them through regular monitoring
Validates logic and data with practices that include data profiling and automated validation tests
Good to have:
Experience in batch scheduling & rationalization through Control M
Exposure working in Snowflake
Exposure to reporting tools like PowerBI/ Tableau etc.
About Company:
