Sr. Databricks Engineer (PySpark and Data Lake) Job at Cloudious LLC, Wilmington, NC

RGd1alZWZ2pvSXl5RTBpZm1NeHRzYnBvOXc9PQ==
  • Cloudious LLC
  • Wilmington, NC

Job Description

Sr. Databricks Engineer (PySpark and Data Lake) Exp - 5 Positions

Location: Wilmington, DE 5 Days onsite role

Long Term Project

Overview

We are seeking a Data Engineer to lead the modernization of legacy ETL systems by migrating Ab Initio workflows to scalable, modular PySpark pipelines on Databricks. The role involves transforming complex data ecosystems into cloud-native architectures while ensuring data integrity, performance, and reliability.

ETL Modernization & Development

Analyze and migrate legacy ETL workflows from Ab Initio to PySpark-based pipelines

Design and develop scalable data pipelines on Databricks

Refactor monolithic processes into modular, reusable components

Leverage existing enterprise datasets to avoid redundancy

Data Integration & Processing

Build and maintain ETL/ELT pipelines integrating data from Snowflake and other sources

Process and publish enriched datasets for downstream applications

Support batch and near real-time data processing

Data Lineage & Optimization

Create end-to-end data lineage and data flow diagrams

Identify redundancies and drive process consolidation and optimization

Ensure adherence to data governance and quality standards

Testing & Validation

Develop unit, integration, and reconciliation frameworks

Perform dual-run comparisons with legacy systems

Validate outputs in UAT and pre-production environments

Deployment & Operations

Support cutover and migration strategy from legacy systems

Decommission legacy workflows and optimize scheduling (e.g., Control-M)

Develop runbooks, monitoring, and operational documentation

Collaboration

Work with data architects, analysts, and downstream application teams

Coordinate user acceptance testing (UAT/FAT) and stakeholder sign-offs

Job Tags

Similar Jobs

Mid-Cumberland Human Resource Agency

Meals-on-Wheels Program Director Job at Mid-Cumberland Human Resource Agency

Job SummaryLocation: Middle Tennessee (13-county service area)Job Type: Full-TimePay: $35.48/hourOverviewWe are seeking a mission-driven Program Director to lead our Meals-on-Wheels and Senior Dining programs serving older adults across Middle Tennessee. ...

Baylor Scott & White Health

RN ICU Nights Job at Baylor Scott & White Health

 ...Patient Population:? Medical, Surgical, Trauma, Neuro, Cardiac Patient Ratios: 1:2...  ...Intensivist coverage, Strong collaboration with ICU Physicians/Surgeons, Committees/Unit Based...  ...Essential Functions of the Role As an RN, conduct detailed clinical assessments and... 

BJC Healthcare

Part Time Clinical Triage Nurse - Access Center Job at BJC Healthcare

 ...technologies to coordinate patient care, to include multiple software applications, computerized clinical guidelines, specialized telephony equipment, etc. Applies standard work procedures to tasks, such as routine medication refill requests, ordering testing, scheduling... 

Russell Medical

Registered Nurse CVU (PRN 7a-7p) Job at Russell Medical

Registered Nurse CVU (PRN 7a-7p)JOB SUMMARY:Provides professional nursing care through the assessment, treatment, and education of patients in accordance with physicians treatment plans.MINIMUM QUALIFICATIONS:EDUCATION: Associates degree in NursingEXPERIENCE:... 

NBCUniversal

News Photographer/Editor Job at NBCUniversal

NBCUniversal is one of the world's leading media and entertainment companies. We create world-class content, which we distribute across our portfolio of film, television, and streaming, and bring to life through our global theme park destinations, consumer products...