Software Engineer - LLM Applications (Python, Java)
FICO, Work from Home
The role
FICO is hiring a Software Engineer to own the quality, safety, and performance of our AI-powered consumer features. This role combines Python and Java backend engineering with hands-on LLM experience to own how our assistants are prompted, evaluated, guarded, and improved. You will treat AI quality as something we measure and build the tooling our analysts and QA engineers rely on.
What You’ll Contribute
- Own the system prompts, tool descriptions, and evaluation prompts behind Personalized Credit Insights and the AI chat assistant, improving them using production evidence.
- Build and grow the evaluation framework: turn production failures into test cases, run regression suites before every prompt or model change, and report results the business can trust.
- Measure and improve LLM-based evaluator reliability by comparing against human-labeled data and tracking precision, recall, and agreement over time.
- Design, tune, and monitor guardrails, including input and output classifiers, PII redaction, prompt-injection defenses, and financial-advice compliance rules defined with Legal and Compliance.
- Lead model upgrades by re-running evaluations, comparing quality, cost, and latency, and shipping safely through A/B tests.
- Reduce cost and latency with prompt caching, model routing, and context management.
- Build tools for Conversation Analysts and QA engineers, such as transcript viewers, labeling workflows, and dashboards.
- Partner with Analytics, Legal, and Compliance on fairness and bias evaluation of AI outputs.
- Leverage AI coding assistants and agentic workflows (Claude Code, Cursor, GitHub Copilot) to accelerate delivery, with the judgment to review and refine their output.
What We’re Seeking
- Bachelor's or Master's in Computer Science, Engineering, or a related field, with demonstrated professional software engineering experience, strong Python, and the ability to read and contribute to Java (Spring Boot) microservices.
- Proven experience building and operating LLM-powered applications in production, including prompt design, tool calling, structured outputs, and streaming.
- Demonstrated experience building evaluations for LLM systems: test sets, regression suites, and LLM-as-judge approaches, including measuring evaluator quality against human labels.
- Working knowledge of Amazon Bedrock or a comparable managed LLM platform, and AWS services such as S3, Lambda, and DynamoDB.
- Proficient SQL and comfort analyzing production data in a data warehouse such as Amazon Redshift.
- Production ownership, including code review, CI/CD, release management, and on-call support, within Agile/Scrum teams.