Lead Data Engineer

POSITION SUMMARY: 

We are seeking an experienced Lead Data Engineer to join the Data Governance team. The Lead Data Engineer for the Enterprise Data Hub Team will collaborate with multiple stakeholders to develop data objects and pipelines on the AWS Cloud Platform. Additionally, they will work with other IT teams on data solution projects, delivering products that integrate various system data sources and harmonize processes and data models. The Lead Data Engineer will oversee the completion of data and business intelligence (BI) related projects and will collaborate with other teams within the organization.

ESSENTIAL FUNCTIONS: 

• Build and maintain orchestration workflows to automate data pipeline execution, scheduling, and monitoring.
• Collaborate with data analysts, and business stakeholders to understand their data needs and translate them into technical solutions.
• Contribute to the development and improvement of data engineering processes and standards.
• Optimize data pipelines for performance, reliability, and cost-effectiveness, leveraging AWS best practices and cloud-native technologies.
• Design and implement robust and scalable data architectures on the AWS platform, including data lakes, data warehouses, and data pipelines.
• Develop and maintain ETL/ELT processes using AWS services such as AWS Glue, AWS Lambda and AWS Step Functions.
• Build and manage data pipelines for ingesting, transforming, and loading data from various sources, including databases, APIs, and cloud storage.
• Collaborate with analysts, and business stakeholders to understand data requirements and deliver scalable data solutions.
• Ensure data quality, security, and compliance with industry best practices and regulatory requirements.
• Troubleshoot and resolve data-related issues, including performance bottlenecks and data integrity problems.
• Mentor junior data engineers and provide guidance on best practices and technologies.
• Stay up-to-date with the latest advancements in AWS technologies and data engineering best practices.

EXPERIENCE, SKILLS AND EDUCATION REQUIREMENTS: 

• Minimum 7 years of experience building and deploying large-scale data processing pipelines in a production environment.
• Hands-on experience in designing and building data pipelines on AWS cloud infrastructure.
• Strong proficiency in AWS services such as Amazon S3, AWS Glue, AWS Lambda, Amazon Redshift, AWS Step Functions etc.
• Strong experience with Apache Spark for data processing and analytics.
• Hands-on experience on orchestrating and scheduling data pipelines using AppFlow, Event Bridge and Lambda.
• Solid understanding of data modeling, database design principles, and SQL and Spark SQL.
• Experience with version control systems (e.g., Git) and CI/CD pipelines.
• Excellent communication skills and the ability to collaborate effectively with cross-functional teams.
• Strong problem-solving skills and attention to details

SALARY RANGE:

The salary range for this position is $150,000.00 to $170,000.00. Where an employee or prospective employee is paid within this range will depend on, among other factors, actual ranges for current/former employees in the subject position; market considerations; budgetary considerations; tenure and standing with the company (applicable to current employees); as well as the employee s/applicant s background, pertinent experience, and qualifications.

BENEFITS OF JOINING IPG:

One of our primary goals is to support the health and well-being of you and your family. Our compensation plan includes the following benefits, in addition to many others:

• Healthcare Options
  • Medical
  • Dental
  • Vision
  • Prescription
• Dependent and Health Care Flexible Spending Accounts
• 401(k) savings plan with company match
• Flexible based Paid Time Off
• Employee Assistance Program
• Legal Assistance Plan
• Tuition Reimbursement
• Employee Stock Purchase Plan
• Exclusive discounts on cell phones, gyms, and everyday purchases

IPG is an equal opportunity employer and we value diversity inclusion and equity at our company. We provide equal employment opportunities to all employees and applicants for employment without regard to race, color, religion, creed, national origin, gender, sexual orientation, age, gender identity, marital status, citizenship, disability, genetic information or veteran status or any other basis prohibited by applicable federal, state or local law.