Sr. Databricks Engineer (PySpark and Data Lake) Exp - 5 Positions
Location: Wilmington, DE 5 Days onsite role
Long Term Project
Overview
We are seeking a Data Engineer to lead the modernization of legacy ETL systems by migrating Ab Initio workflows to scalable, modular PySpark pipelines on Databricks. The role involves transforming complex data ecosystems into cloud-native architectures while ensuring data integrity, performance, and reliability.
ETL Modernization & Development
Analyze and migrate legacy ETL workflows from Ab Initio to PySpark-based pipelines
Design and develop scalable data pipelines on Databricks
Refactor monolithic processes into modular, reusable components
Leverage existing enterprise datasets to avoid redundancy
Data Integration & Processing
Build and maintain ETL/ELT pipelines integrating data from Snowflake and other sources
Process and publish enriched datasets for downstream applications
Support batch and near real-time data processing
Data Lineage & Optimization
Create end-to-end data lineage and data flow diagrams
Identify redundancies and drive process consolidation and optimization
Ensure adherence to data governance and quality standards
Testing & Validation
Develop unit, integration, and reconciliation frameworks
Perform dual-run comparisons with legacy systems
Validate outputs in UAT and pre-production environments
Deployment & Operations
Support cutover and migration strategy from legacy systems
Decommission legacy workflows and optimize scheduling (e.g., Control-M)
Develop runbooks, monitoring, and operational documentation
Collaboration
Work with data architects, analysts, and downstream application teams
Coordinate user acceptance testing (UAT/FAT) and stakeholder sign-offs
...QUALIFICATIONS: To perform this job successfully, an individual must have or be working towards the following qualifications: Class B CDL Strong knowledge of safety procedures and regulations. Excellent communication skills. Flexibility with working hours...
...newspaper of the year in 2024, is owned by Newspapers of New England, a family-owned company with a commitment to serious local journalism.We offer competitive pay, paid vacation and personal days and benefits, including health, dental and vision insurance and a 401K...
...Onboarding, and Simeio Identity Orchestrator. The company has been recognized for its business and technical leadership and highly rated by Gartner, Forrester, and KuppingerCole, and was ranked by Great Places to Work. For more information visit simeio.com Simeio is an equal...
...is noticed and valued every day. Financial Analyst II Are you a recent graduate looking to build a strong foundation in corporate finance and financial planning and analysis? Would you like the opportunity to support strategic initiatives including mergers, acquisitions...
...Differential ~ Benefits Eligible BJC RN Career Ladder - The BJC RN Career Ladder differentiates BJC as the place for nurses to work in the greater St. Louis area.... ...services, an infusion center, a wound care clinic, and the Huckstep Heart and Lung Center. Bonne...