How to set up a user feedback loop for your application

Choose your feedback signals

Feedback comes in two forms. Explicit feedback is a rating the user gives on purpose, like a thumbs up or down or a star rating: unambiguous, but rare and skewed toward unhappy users. Implicit feedback is derived from what users do, like retrying a query or editing a draft: you have data on every trace, but it needs interpretation.

The best signals depend heavily on your use case. Some examples as inspiration:

Tips:

• signals combine well, no need to implement only one
• capture free-form text wherever it fits (can be as a comment on the score), as much of this analysis is now done by agents, and they work better with more context
• the academy examples show how signals are chosen for a specific application and change over its lifecycle

Capture the signals as scores

Every signal ends up as a score on the trace that produced the output. A score has a name, a value with a data type (boolean, numeric, categorical or free text), and an optional comment. Evaluators also store their results as scores, so your user feedback can land in the same filters, dashboards, and analytics.

The route into that score depends on where the signal originates:

import { LangfuseBrowserClient } from "@langfuse/browser";

const langfuse = new LangfuseBrowserClient({
  publicKey: process.env.NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY!,
});

// On a thumbs up / down click
await langfuse.score({
  traceId, // returned by your backend with the response
  id: `response_rating-${traceId}`, // stable id, so re-rating updates instead of duplicating
  name: "response_rating", // one descriptive name per signal
  value: 1, // 1 for thumbs up, 0 for thumbs down
  dataType: "BOOLEAN",
});

from langfuse import get_client

langfuse = get_client()

# The user edited the drafted reply before sending it
langfuse.create_score(
    trace_id=trace_id,  # the trace that produced the draft
    name="draft_edited",
    value=1,
    data_type="BOOLEAN",
    comment=edit_diff,  # the edit itself, as context for later analysis
)

import { LangfuseClient } from "@langfuse/client";

const langfuse = new LangfuseClient();

// The user edited the drafted reply before sending it
await langfuse.score.create({
  traceId, // the trace that produced the draft
  name: "draft_edited",
  value: 1,
  dataType: "BOOLEAN",
  comment: editDiff, // the edit itself, as context for later analysis
});

// Flush the scores in short-lived environments
await langfuse.flush();

Act on the feedback

Monitor how quality is trending

Score analytics and custom dashboards chart how each signal moves over time. One caveat: not every signal is a clean quality metric, e.g. thumbs feedback skews toward unhappy users.

Surface interesting traces

Filter traces by score, for example response_rating = 0, to get a concrete list of traces to investigate. Error analysis is a structured way to read and cluster them, and an annotation queue brings in more reviewers.

Use it as input for structured improvement

Add the failing cases to a dataset, test a fix with an experiment before shipping, and turn a recurring failure mode into an automated evaluator that scores every production trace from then on.

Much of this can also be handed to an agent, see Let an agent act on the feedback below.