We prefer candidates who can work from our Carmel-area office on a regular basis. Remote arrangements may be considered for exceptional candidates based on role requirements and business needs.

About Sher Innovations

Sher Innovations builds enterprise software for complex, operations-heavy organizations. We focus on production reliability, governed security, and disciplined delivery. Our teams build platforms and applications that stay up, stay secure, and evolve without chaos.

Company Values: Build Superior People-First Partner Impact.

Engineering Values: Integrity, One Team, Empathy, Ownership, Craftsmanship, Drive.

The Opportunity
At Sher Innovations, your work runs in production, not a lab. Were expanding our engineering capacity to meet demand for reliable, governed systems in operations-heavy environments. Were looking for senior engineers who take end-to-end ownership, ship with discipline, and raise the bar on production readiness.

Were building a team that holds itself and each other to high accountability and a high bar for quality.

About The Role
As a Principal AI/ML Engineer (Full-Stack, Applied AI) at Sher Innovations, you will build end-to-end applications that operators can trust in production. This is not a research-only role. You will ship complete systems across frontend, backend, data, and deployment, embedding AI where it creates real leverage inside the workflow.

You will lead the technical design of AI-enabled capabilities across the stack, including product integration, services, data pipelines, evaluation, model serving, and monitoring. Youll partner closely with engineering leadership and customer stakeholders to define success metrics, validate performance, and deliver production-ready releases.

Youll help set the standard for how we build and operate AI in enterprise and regulated environments: governed, observable, and built for operational reality.

How We Work
We operate in a strategic hybrid model built for high-accountability teams. Most focused work is remote. We use intentional in-person time for architecture reviews, working sessions, and stakeholder alignment.

In This Role, You Will:

Build and Deploy

• Design and deploy LLM-enabled workflows that automate complex operational tasks with clear guardrails.
• Deliver production-grade model serving patterns with reliability, latency, and cost controls.

Engineer the System, Not Just the Model

• Architect scalable data pipelines, feature/embedding workflows, and retrieval patterns where appropriate.
• Establish evaluation harnesses, regression tests, and quality gates for model changes.

Own Production Outcomes

• Implement monitoring for quality, drift, failure modes, and operational performance.
• Drive incident-ready practices: runbooks, rollback paths, and measurable SLOs.

Lead with Clarity

• Set engineering standards for ML systems design, code quality, and operational readiness.
• Mentor through design reviews, code reviews, and pragmatic technical guidance.

Partner with Stakeholders

• Work directly with customer and partner stakeholders to translate requirements into shipped outcomes.
• Communicate tradeoffs clearly, especially around risk, safety, cost, and reliability.

The Profile: What You Will Bring

We are seeking a principal-level engineer who operates with a leadership mindset and strong ownership:

• Production-first ML mindset: You build with evaluation, monitoring, and failure modes in mind.
• Systems thinking: You design end-to-end architecture across data, models, services, and interfaces.
• Pragmatic execution: You choose approaches that are maintainable and operable, not just impressive.
• Ownership: You take responsibility through delivery and beyond.
• Communication: You can explain technical decisions clearly to engineers and non-engineers.

Qualifications:

Required:

• 8+ years building production software systems, including ownership of system design and delivery.
• Proven experience deploying ML/LLM-enabled capabilities in production environments.
• Strong Python experience and comfort designing backend services and APIs.
• Experience with evaluation and monitoring (quality, drift, performance, cost) for ML systems.
• Strong engineering fundamentals: testing, observability, and operational readiness.

Preferred:

• Cloud/infrastructure production experience with GCP, AWS, Azure, or Linode
• Infrastructure and deployment experience with Terraform, Kubernetes, Docker, and VMs
• Backend framework experience with Phoenix, FastAPI, or Django (or equivalent)
• Data systems experience with relational and NoSQL, plus Redis
• Streaming or real-time systems experience with Kafka, MQTT, or similar
• Network and traffic management familiarity (load balancing, HAProxy, etc.)
• Mobile experience (Swift, Kotlin, or React Native)
• Experience in regulated or operations-heavy domains

Education:

• Bachelors or Masters in Computer Science (or equivalent professional experience)

Benefits & Culture

We are committed to fostering a diverse and inclusive workplace where every employee feels valued and supported.

• Competitive Compensation: Salary + Equity
• Health & Wellness: Medical, Dental, and Vision
• Future Planning: 401(k)
• Flexibility: Unlimited PTO and Strategic Hybrid work model
• Perks: Daily lunches and professional development opportunities

Sher Innovations is an equal opportunity employer.