Senior Databricks Engineer

Jersey City, NJ — Hybrid (3 Days Onsite) Contract to Hire W2 Only — No Third Parties 1 8+ Years Experience Azure Databricks, Python, PySpark, SQL This role is responsible for designing, developing, and deploying cloud data solutions as part of a broader cyber data initiative focused on building out a security and risk data platform for Information Security. The engineer will develop data feeds and contribute to the buildout of a Cybersecurity Data Lake — centralizing data and establishing effective data governance around data sources and lineage using Databricks. This individual will collaborate with developers, data owners, governance leads, and business analysts across IT and aligned application stakeholders. ? Critical Requirements Financial services or similarly regulated environment experience strongly preferred Responsibilities Design, develop, test, and support cloud data solutions focusing on data ingestion, data quality, tuning, and performance of upstream cybersecurity data sources using DatabricksProvide indirect leadership and mentorship to junior team members; lead data feed development efforts and design initiativesParticipate in development meetings to align priorities and objectives, assign tasks, and share experiences across applications under developmentConsult with technology and development teams to coordinate integration of applications within the broader software ecosystemCapture and document metadata for Key Data Elements (KDEs) to ensure accuracy and completeness for Data Quality (DQ) rules and daily dataset processingWork with the data architecture team to align KDEs to logical data models, develop physical data structures, and document physical data names, definitions, and typesPartner with data owners and stakeholders to create technical requirements and DQ rules around data elements needed in the data warehousePartner with Business Management and Data Owners to identify critical metrics and data fields required for metric dashboardsEstablish views to encapsulate data for downstream consumption and ensure alignment with DQ rules established on metadataUtilize IBM TWS production job scheduling and adhere to standards around daily scheduling and batch monitoring of production jobsIdentify and resolve data quality issues including inaccuracies and incomplete information; implement improved procedures and processes Qualifications & Skills Azure & Cloud Platform Azure Databricks & Unity Catalog Azure Data Factory Azure Data Lake Storage Azure Functions Azure Event Grid Azure Log Analytics & Monitor Development & Data Engineering Python / Java SQL Server, Oracle / PL-SQL, T-SQL ETL / ELT REST API & Microservices CI/CD — Git, Jenkins, Azure DevOps Collibra DQIM/DQE, IBM Infosphere DA Security Domain & Compliance Strong knowledge of enterprise Information Security data domains: Phishing, Identity Management, Privileged Access, Cloud Security, Incident Response, Vulnerability Management, and Threat DetectionData knowledge of PaaS/SaaS products including ServiceNow, CrowdStrike, MS Purview, Proofpoint, WIZ.IO, JIRA, SharePoint, and Azure Active DirectoryUnderstanding of information security frameworks (NIST, CIS, CRI Profile) and regulatory compliance (NYSDFS, GDPR, CCPA) ? Nice to Have Knowledge of Microsoft Sentinel for SIEMKnowledge of IBM Tivoli Workload SchedulerExposure to Power BI for data visualization and reporting