Ai Driven Data Profiling Engineer

Requirement: AI-Driven Data Profiling Engineer

Location: NYC, NY

The Role

We are seeking an AI-Driven Data Profiling Engineer to support a Proof of Concept focused on profiling MGA BDX data through an autonomous agent workflow.

The successful candidate will work closely with data profiling specialists, data consumers, onboarding teams, and enterprise data stakeholders to refine agent outputs and accelerate the onboarding of MGA programs into the BDX management platform and enterprise data hub.

The role will involve refining this autonomous workflow, improving the quality and usability of profiling outputs, and ensuring that results meet the needs of downstream data consumers.

Key Responsibilities

• Refine and enhance the autonomous agent workflow for MGA BDX data profiling.
• Work with data profiling teams and business data consumers to define profiling requirements and expected outputs.
• Review and improve the behavior, prompts, instructions, controls, and outputs of the Kiro agents.
• Support discovery and extraction of PMA-related information from MGA BDX data.
• Develop and refine profiling plans for individual MGA programs.
• Run the profiling workflow across a backlog of approximately 80 100 MGA programs.
• Identify recurring data quality issues, structural variations, missing fields, anomalies, and business-rule exceptions.
• Continuously improve the workflow based on findings from each profiling cycle.
• Validate profiling results through reviewer-agent and human review processes.
• Ensure profiling outputs are clear, actionable, traceable, and suitable for downstream teams.
• Provide profiling results to teams responsible for onboarding data into:
  • The BDX management platform
  • The enterprise data hub
• Maintain documentation of profiling rules, discovered patterns, exceptions, decisions, and workflow changes.
• Help define dashboards, metrics, and reporting requirements for profiling progress and data quality.
• Collaborate with business, data, engineering, and platform stakeholders to resolve issues and clarify requirements.
• Contribute to recommendations regarding the potential wider adoption of an autonomous-agent approach.

Required Skillsets and Experience

• Strong background in the insurance industry.
• Experience working with Managing General Agents (MGAs) and/or insurance distribution data.
• Understanding of BDX data, bordereaux, program data, and insurance data onboarding processes.
• Familiarity with insurance data structures, policy, premium, claims, exposure, and related data concepts.
• Experience working with data used for insurance operations, reporting, underwriting, finance, or enterprise data platforms.
• Strong data analysis and data profiling experience.
• Experience identifying data quality issues, anomalies, inconsistencies, missing values, duplicates, and structural variations.
• Ability to interpret complex, semi-structured, and inconsistent data sources.
• Experience defining profiling rules, validation logic, business rules, and data quality metrics.
• Ability to translate profiling results into clear recommendations for technical and business audiences.
• Strong understanding of data lineage, metadata, mappings, data contracts, and onboarding requirements.
• Experience working with data consumers to refine requirements and improve the usefulness of data outputs.
• Proficiency in agentic engineering and autonomous workflow design.
• Hands-on experience designing, configuring, testing, and refining AI-agent workflows.
• Experience with below Kiro agents
  • Discovery Agent
  • PMA Extraction Agent
  • Planning Agent
  • Profiling Agent
  • Reviewer Agent
  • Dashboard Updater Agent
  • Other generative AI, coding-agent, or orchestration tools
• Ability to design effective prompts, agent instructions, task decomposition, validation steps, and review mechanisms.
• Experience creating agent workflows that include planning, execution, review, exception handling, and iterative improvement.
• Understanding of human-in-the-loop controls and approaches for ensuring reliable and auditable AI outputs.
• Ability to troubleshoot agent behavior and improve consistency, accuracy, and explainability.