Principal AI Engineer

Principal AI Engineer – Agentic AI / LLM

Location: New York City, NY

Work Arrangement: Hybrid – 3 Days Onsite per Week

Employment Type: Full-Time

Work Hours: Must overlap with EST

Experience: 8–14 Years (10+ Years Preferred)

Visa: H-1B Transfer Candidates Can Be Considered

About the Role:

We are seeking a highly experienced Principal AI Engineer – Agentic AI / LLM to lead the architecture, development, and production deployment of enterprise-scale Agentic AI and Generative AI solutions for Fortune 500 clients.

This is a highly hands-on engineering and technical leadership position requiring strong expertise in Python, Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Model Context Protocol (MCP), AI agent orchestration, and cloud infrastructure.

The ideal candidate will have a strong software engineering foundation and proven experience designing, building, and deploying multiple production-grade AI agent systems.

You will own the complete AI engineering lifecycle, from architecture and data pipelines to backend services, agent orchestration, evaluation frameworks, cloud deployment, performance optimization, and ongoing production reliability.

Candidates should demonstrate actual production ownership of AI systems rather than experience limited to research, proof-of-concepts, or basic LLM integrations.

Key Responsibilities

\n
• Architect, design, and develop production-grade Agentic AI systems using LLMs, RAG, tool calling, and agent orchestration.
\n
• Build autonomous AI agents capable of reasoning, retrieval, planning, and executing secure enterprise actions.
\n
• Own end-to-end AI system development, including data pipelines, backend services, agent orchestration, evaluation, observability, and cloud infrastructure.
\n
• Design and implement advanced RAG pipelines covering document ingestion, chunking strategies, embeddings, vector search, hybrid retrieval, reranking, and response validation.
\n
• Build and integrate MCP (Model Context Protocol) servers and secure tool-calling interfaces to connect AI agents with enterprise applications and business systems.
\n
• Develop multi-agent workflows using LangGraph, Google ADK, CrewAI, Claude Agent SDK, or equivalent frameworks.
\n
• Implement agent state management, planning, reasoning, tool execution, and human-in-the-loop workflows.
\n
• Build AI guardrails, validators, policy checks, deterministic fallbacks, self-correction loops, circuit breakers, and rollback strategies.
\n
• Develop LLM evaluation frameworks, including offline evaluation datasets, automated testing, continuous evaluation, and regression detection.
\n
• Implement LLM observability, tracing, and monitoring using LangSmith, Langfuse, or equivalent platforms.
\n
• Optimize AI applications for token consumption, context-window utilization, latency, cost, throughput, and scalability.
\n
• Design and maintain scalable backend services, APIs, microservices, and distributed systems.
\n
• Develop and maintain high-quality, production-ready Python and SQL code.
\n
• Deploy and operate AI applications using AWS, Azure, or GCP.
\n
• Implement CI/CD pipelines and infrastructure-as-code using Terraform or CloudFormation.
\n
• Collaborate with product, engineering, data science, and business stakeholders to translate business requirements into scalable technical solutions.
\n
• Provide technical leadership, conduct architecture and code reviews, establish engineering standards, and mentor engineering teams.
\n

Required Qualifications:

1. Software Engineering & Technical Foundation

\n
• 8–14 years of professional software engineering experience, with 10+ years preferred.
\n
• Strong hands-on experience developing large-scale applications using Python.
\n
• Strong understanding of data structures, algorithms, and software engineering fundamentals.
\n
• Experience designing and developing APIs, microservices, and distributed systems.
\n
• Hands-on experience building and operating production-grade data pipelines.
\n
• Strong knowledge of system architecture, scalability, and performance optimization.
\n
• Experience with at least one additional backend or systems programming language, such as Go, Rust, Java, C, or C++, is highly desirable.
\n

2. LLM & Generative AI Engineering

\n
• 2+ years of hands-on LLM/Generative AI engineering experience.
\n
• Experience designing, developing, and deploying production-grade LLM applications.
\n
• Demonstrated experience building and taking multiple Agentic AI systems into production.
\n
• Strong understanding of LLM architecture, prompt engineering, context management, and model behavior.
\n
• Experience integrating modern LLM APIs and enterprise AI platforms.
\n

3. Agentic AI & Frameworks

Hands-on production experience with one or more of the following:

\n
• LangGraph
\n
• Google ADK
\n
• CrewAI
\n
• Claude Agent SDK
\n
• Other equivalent production-grade agent frameworks
\n

Strong understanding and implementation experience in:

\n
• Agent orchestration and state management.
\n
• Planning and reasoning workflows.
\n
• Tool execution and function calling.
\n
• Multi-agent architectures and workflows.
\n
• Human-in-the-loop patterns.
\n
• Multi-step autonomous AI workflows.
\n
• Agent reliability, error handling, and failure recovery.
\n

4. MCP & Enterprise Tool Integration

\n
• Strong hands-on experience with Model Context Protocol (MCP).
\n
• Experience building and deploying MCP servers.
\n
• Experience developing secure tool interfaces for AI agents.
\n
• Understanding of function calling, tool calling, and API integrations.
\n
• Experience securely connecting AI agents with enterprise applications and business systems.
\n

5. RAG, Embeddings & Vector Databases

Strong understanding of RAG architecture from first principles, including:

\n
• Document ingestion and processing.
\n
• Chunking strategies and optimization.
\n
• Embeddings generation.
\n
• Vector search and semantic retrieval.
\n
• Hybrid retrieval.
\n
• Reranking techniques.
\n
• Retrieval optimization.
\n
• Context construction and management.
\n
• Response validation and grounding.
\n

Experience with one or more vector databases:

\n
• Pinecone
\n
• Milvus
\n
• Weaviate
\n
• FAISS
\n
• OpenSearch
\n
• Other vector databases or cloud equivalents.
\n

6. LLM Evaluation, Observability & Reliability

\n
• Experience building LLM evaluation frameworks.
\n
• Offline evaluation datasets and automated evaluation.
\n
• Continuous/online evaluation and regression testing.
\n
• AI quality measurement and performance monitoring.
\n
• LLM observability, tracing, and instrumentation.
\n
• Experience with LangSmith, Langfuse, or equivalent platforms.
\n
• Experience implementing AI guardrails, validators, fallback mechanisms, and reliability patterns.
\n

7. Cloud Infrastructure & DevOps

Strong AWS experience is preferred, including:

\n
• Amazon EKS / ECS
\n
• AWS Lambda
\n
• Amazon S3
\n
• DynamoDB
\n
• Amazon Redshift
\n
• AWS Step Functions
\n
• Amazon Bedrock
\n

Additional experience with:

\n
• Azure or GCP.
\n
• Docker and Kubernetes.
\n
• Terraform or CloudFormation.
\n
• CI/CD pipelines.
\n
• Infrastructure-as-code.
\n
• Cloud deployment and production operations.
\n

8. AI Performance & Cost Optimization

Experience optimizing production LLM and Agentic AI systems through:

\n
• Token consumption optimization.
\n
• Context-window management.
\n
• Context shaping, pruning, and compaction.
\n
• Model routing.
\n
• Caching strategies.
\n
• Batching and streaming.
\n
• Parallel tool calls.
\n
• Latency optimization.
\n
• Cost and throughput optimization.
\n
• Performance testing against defined service-level objectives (SLOs).
\n

Preferred Qualifications

\n
• Experience with Amazon Bedrock, Azure AI, or Google Vertex AI.
\n
• Experience building enterprise-scale AI platforms.
\n
• Experience working in enterprise or regulated environments.
\n
• Strong knowledge of scalable distributed systems and backend architectures.
\n
• Strong analytical, problem-solving, and debugging skills.
\n
• Ability to translate ambiguous business requirements into technical architecture.
\n
• Strong client-facing communication and stakeholder management skills.
\n
• Demonstrated technical leadership, architecture ownership, and mentoring experience.
\n