Job DescriptionJob DescriptionWe're looking for a Senior AI Engineer to join Procore's Evals & Goldsets team, which measures and improves the quality of the AI agents we ship to the construction industry. You'll build and run the systems that tell our AI teams how well their agents are doing: production tracing in Langfuse and Databricks, our online LLM judge, and the goldsets and benchmarks behind our offline evals.\n As a Senior AI Engineer, you will partner with data and ML engineers to make our evals faster, cheaper, more trustworthy, and harder to saturate, and you'll turn production traces into feedback loops that improve agent behavior. We're a small, fast-moving team that pairs hands-on building with architecture and review, and we value engineers who are rigorous about measurement and comfortable with ambiguity. If you want to shape how AI in construction gets tested and improved, come join us.\n\nThis hybrid position reports to the Engineering Manager for Evals & Goldsets and will be based in our Austin, TX office.\n\nWhat you’ll do: \n\n Improve our online evals: LLM judge accuracy, calibration, and the infrastructure that runs it in production.\n Build and maintain production trace infrastructure with Langfuse and Databricks.\n Build and maintain data pipelines for the goldsets behind our evals, including annotation and quality checks.\n Build agentic loops that mine production traces for failures and turn them into new eval cases and agent improvements.\n Improve the runtime, cost, and reliability of our online and offline eval pipelines.\n Contribute code and architecture, review designs, and mentor other engineers on the team.\n\n\nWhat we’re looking for: \n\n 5+ years of experience in software, AI, or ML engineering.\n Strong proficiency in Python; experience with TypeScript is a plus.\n Experience evaluating LLM systems, such as LLM-as-judge, rubric design, goldset curation, or measuring judge agreement.\n Experience with LLM observability or evaluation tools such as Langfuse, Braintrust, or similar.\n Experience building data pipelines, ETL processes, or streaming data systems.\n Experience with analytics platforms and data warehouses such as Databricks or Snowflake.\n Ability to communicate eval results clearly and influence the teams that build on them.\n Degree in computer science or a related technical field, or equivalent practical experience.\n\nAdditional Information\n\nBase Pay Range:140,960.00 - 193,820.00 USD Annual\n\nThis role may also be eligible for Equity Compensation and/or Bonus Incentive Compensation. Procore is committed to offering competitive, fair, and commensurate compensation. Actual compensation will be based on a candidate’s job-related skills, experience, education or training, and location.\nFor Los Angeles County (unincorporated) Candidates:\nProcore will consider for employment all qualified applicants, including those with arrest or conviction records, in accordance with the requirements of applicable federal, state, and local laws, including the City of Los Angeles’ Fair Chance Initiative for Hiring Ordinance, the Los Angeles County Fair Chance Ordinance for Employers, and the California Fair Chance Act.\n\nA criminal history may have a direct, adverse, and negative relationship on the following job duties, potentially resulting in the withdrawal of the conditional offer of employment: 1. appropriately managing, accessing, and handling confidential information including proprietary and trade secret information, as well as accessing Procore's information technology systems and platforms; 2. interacting with and occasionally having unsupervised contact with internal/external customers, stakeholders, and/or colleagues; and 3. exercising sound judgment.\n
Click on the below icons to share this job to Linkedin, Twitter!
Job DescriptionJob DescriptionWe're looking for a Senior AI Engineer to join Procore's Evals & Goldsets team, which measures and improves the quality of the AI agents we ship to the construction industry. You'll build and run the systems that tell our AI teams how well their agents are doing: production tracing in Langfuse and Databricks, our online LLM judge, and the goldsets and benchmarks behind our offline evals.\n As a Senior AI Engineer, you will partner with data and ML engineers to make our evals faster, cheaper, more trustworthy, and harder to saturate, and you'll turn production traces into feedback loops that improve agent behavior. We're a small, fast-moving team that pairs hands-on building with architecture and review, and we value engineers who are rigorous about measurement and comfortable with ambiguity. If you want to shape how AI in construction gets tested and improved, come join us.\n\nThis hybrid position reports to the Engineering Manager for Evals & Goldsets and will be based in our Austin, TX office.\n\nWhat you’ll do: \n\n Improve our online evals: LLM judge accuracy, calibration, and the infrastructure that runs it in production.\n Build and maintain production trace infrastructure with Langfuse and Databricks.\n Build and maintain data pipelines for the goldsets behind our evals, including annotation and quality checks.\n Build agentic loops that mine production traces for failures and turn them into new eval cases and agent improvements.\n Improve the runtime, cost, and reliability of our online and offline eval pipelines.\n Contribute code and architecture, review designs, and mentor other engineers on the team.\n\n\nWhat we’re looking for: \n\n 5+ years of experience in software, AI, or ML engineering.\n Strong proficiency in Python; experience with TypeScript is a plus.\n Experience evaluating LLM systems, such as LLM-as-judge, rubric design, goldset curation, or measuring judge agreement.\n Experience with LLM observability or evaluation tools such as Langfuse, Braintrust, or similar.\n Experience building data pipelines, ETL processes, or streaming data systems.\n Experience with analytics platforms and data warehouses such as Databricks or Snowflake.\n Ability to communicate eval results clearly and influence the teams that build on them.\n Degree in computer science or a related technical field, or equivalent practical experience.\n\nAdditional Information\n\nBase Pay Range:140,960.00 - 193,820.00 USD Annual\n\nThis role may also be eligible for Equity Compensation and/or Bonus Incentive Compensation. Procore is committed to offering competitive, fair, and commensurate compensation. Actual compensation will be based on a candidate’s job-related skills, experience, education or training, and location.\nFor Los Angeles County (unincorporated) Candidates:\nProcore will consider for employment all qualified applicants, including those with arrest or conviction records, in accordance with the requirements of applicable federal, state, and local laws, including the City of Los Angeles’ Fair Chance Initiative for Hiring Ordinance, the Los Angeles County Fair Chance Ordinance for Employers, and the California Fair Chance Act.\n\nA criminal history may have a direct, adverse, and negative relationship on the following job duties, potentially resulting in the withdrawal of the conditional offer of employment: 1. appropriately managing, accessing, and handling confidential information including proprietary and trade secret information, as well as accessing Procore's information technology systems and platforms; 2. interacting with and occasionally having unsupervised contact with internal/external customers, stakeholders, and/or colleagues; and 3. exercising sound judgment.\n
yes
Send Us Your Feedback
Sign
In
Please check your email. We have sent you a password reset link.
This link will expire in one hour.
Please input your account's email
Apply to Senior AI Engineer
Procore • Austin, Texas
✓
Application Started!
Redirecting you to complete your application...
Get $5 added to your wallet for registering an account. Refer others and get $1 for each successful referral. Earn while you search for jobs and redeem in gift cards!Register to search from millions of jobs across the United States. Upload your resume and get matched with top employers today!
Welcome to SnapRecruit! Your account has been created via Google.
Get $5.00 Added to Your Wallet!
Upload your resume below and apply to this job to have $5.00 added directly to your wallet, activate your account, and qualify for high-priority employer matches.
Perks & Rewards You Receive with SnapRecruit:
$5.00 Resume Bonus
Added automatically to your wallet when you apply with your resume.
$1.00 Per Referral
Earn $1.00 cash rewards for every friend who joins SnapRecruit in the US.
Earn While You Search
Get credited $0.02 - $0.05 every time you explore sponsored job listings.
Amazon eGift Cards
Redeem your earned wallet funds for Amazon eGift Cards delivered to your email.
Get $5 added to your wallet for registering anRegister to search from millions of jobs across the United States. Upload your resume and get matched with top employers today!
account.
Refers others and get $1 for each successful referral.
Earn while you search for jobs and redeem in
gift cards!
Upload your resume for a free professional assessment
Loading, Please wait..!!
Send this job to my email
Please wait..!!
Confirmation
Are you sure?
List of resumes
Confirmation
It is working for Chrome Mozilla only for now
We use essential cookies to keep SnapRecruit secure and reliable. With your permission, we also use optional analytics, AI support chat, and push-alert cookies to improve your job search experience. You can change or withdraw your choice anytime.
Privacy Policy •
Cookie Policy •
Your Privacy Rights