Did an experiment on improving LLM accuracy with Structured Generation using my stuff: https://www.linkedin.com/posts/jarnomontonen_improving-llm-accuracy-with-levlo-languages-activity-7223648019051626497-Ow0i

Cool stuff. Reminds me of https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#constrained-output-with-grammars . I've used it to generate some scenarios for RPG games with friends. One issue that I've encountered (ignoring overabundance of tropes) was that the model was sometimes very uncooperative and refused to progress through the given template. Instead it got stuck in some place where it was free to generate arbitrary text and just happily stuttered around, never reaching the end. Have you encountered this? If so - what's your solution?

Thanks and interesting use-case! Yes, there are several solutions for structured/constrained/guided generation, but all of the ones I've seen base the structure validation either on regex or context-free grammars. I can do context-sensitive grammars and semantic rules. Say you want to generate RPG character attribute values between 4-18, but the sum should be based on the level of the character. I don't think the solutions I've seen offer good support for something like that, but it would be fairly trivial with Levlo. I haven't really tried structures with free arbitrary text beyond just generated names, but I would imagine either prompting the LLM to only generate arbitrary text of certain length (as in "a paragraph", "couple sentences") or using appropriate length in samples would work. Have you tried? I suppose a potential challenge could be to make the LLM understand which part of the structure you mean with the length instructions in prompt, but I would be surprised if a dozen or so samples wouldn't work. Although, maybe the samples would end up directing the content of the arbitrary text too much as well..

Actually some better prompting might have been the way to go... I was rewriting the whole scenario with LLM over many iterations (leaving the PC overnight) and sometimes it was randomly looping somewhere in the middle of text. I guess that shorter, maybe section-by-section prompts could prevent that... I've tried the grammar-based approach to limit the length but the issue there was that the model sometimes planned out the text for more sentences than the grammar permitted. This resulted in truncated descriptions.

but the issue there was that the model sometimes planned out the text for more sentences than the grammar permitted

Yeah I can see this being a problem