Category: LLMs

LLMs still cannot detect bad UX

Post author By nielstron
Post date October 8, 2026
Categories In LLMs
No Comments on LLMs still cannot detect bad UX

TLDR; I gave 4 coding agents a $100 budget to build a functional PDF editor. The implementations are okay, but not great: within a few clicks I find bugs in almost all of them. I hypothesize that this is partly because agents don’t interact with software like humans: they don’t pursue concrete goals and experience […]

Leveraging Constrained Sampling for Fill-In-the-Middle Code Completion

Post author By nielstron
Post date August 14, 2024
Categories In LLMs
No Comments on Leveraging Constrained Sampling for Fill-In-the-Middle Code Completion

Existing constrained sampling methods only focus on generating code that will be a syntactically well-formed program on its own after completion. What if we have some remaining code suffix and want the entirety to become a valid program? In this post I explore how to leverage existing tooling to create well-formed Copilot-like Fill-In-the-Middle (FIM) code […]

Proper Well-Formedness Guarantees for Finite LLM Sampling

Post author By nielstron
Post date August 13, 2024
Categories In LLMs
No Comments on Proper Well-Formedness Guarantees for Finite LLM Sampling

Even under constrained sampling, LLMs can produce syntactically invalid output. In this post I explore how this can happen and how to resolve the issue using results from language theory.