Generate more data without repeating the same mistakes

A language model can produce instructions, answers, critiques, or simulated interactions. Self-Instruct is an early example of building an instruction dataset through model generation and filtering. The generator supplies candidates; the rest of the pipeline decides whether those candidates carry a useful learning signal.

Different wording can conceal the same reasoning template. Track sources, solution mechanisms, required tools, difficulty, and error types. A dataset with many surface variants but one narrow strategy can improve familiar tests while leaving the intended capability gap untouched.

Where possible, check generated answers with executable tests, mathematical review, or independent evidence. Have reviewers inspect a sample of rejected as well as accepted examples. Filtering that rewards fluency can remove unusual valid solutions while retaining confidently expressed mistakes.

Compare an equal training budget against the exact untouched base and a relevant alternative dataset. Include held-out task families, not just variants from the same generator. Report filtering yield and error rates so a claimed dataset improvement can be connected to the examples that actually reached training.