AgentToolDrop

59 of 71

LangSmith

Traces, evaluates and monitors LLM apps and agents on LangChain's hosted platform via Python and JS SDKs

Install the Python SDK

pip install -U langsmith
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=ls_...

Copied from the project's docs at github.com/langchain-ai/langsmith-sdk

Why you'd want it

LangSmith is LangChain's platform for debugging, evaluating and monitoring language model apps and agents; the MIT-licensed Python and JavaScript SDKs send traces to it, for example by wrapping the OpenAI client. It works with any LLM application and has a native integration with LangChain Python and LangChain JS. There is a free Developer plan with a monthly trace allowance and paid per-seat plans, with pay-as-you-go usage beyond the included traces.

Best for

  • Teams already building with LangChain
  • Hosted tracing without running infrastructure
  • Python and JS/TS apps

Use it when

You want managed tracing and evals with no servers to run, especially if your app already uses LangChain.

Skip it if

You want to self-host without an enterprise contract; self-hosted and hybrid deployment are Enterprise plan options. Usage beyond the included traces is billed pay-as-you-go.

Editor's note

The GitHub repo holds the client SDKs; the LangSmith platform itself is a hosted product.

Related tools

Same category or shared tags, by drop score.

  1. Rank 5

    Langfuse

    Evals

    Traces LLM calls and agent steps, then runs evals, datasets and prompt versioning on top of them

    • score 85
    • 35.8k stars
    • pushed
    • v4.59.0 on
    • MIT

    Install the Python SDK

    pip install langfuse
    Use it when, skip it if, stats for Langfuse

    Why you'd want it

    Langfuse instruments an app through Python and JS/TS SDKs or integrations such as the OpenAI SDK, LangChain, LlamaIndex, Vercel AI SDK and Mastra, and records LLM calls, retrieval and agent actions as traces you can inspect. The same platform covers prompt management with server and client caching, datasets for pre-deployment tests, and evaluations via LLM-as-a-judge, code evaluators, user feedback and manual labeling. You can use Langfuse Cloud or self-host it with Docker Compose, on a VM or on Kubernetes via Helm.

    Best for

    • Teams that want to self-host LLM observability
    • Tracing, prompts and evals in one place
    • Python and JS/TS stacks

    Use it when

    You need to see what your LLM app or agent did on each request and want to evaluate and iterate on prompts against the same traces.

    Skip it if

    You do not want to run a server stack yourself and cannot use a hosted account; self-hosting means running the Docker Compose or Kubernetes deployment. Code under the ee/ directories is not MIT licensed.

    Stats

    Stars
    35,832
    Last push
    Release
    v4.59.0,
    License
    MIT
    Pricing
    Open source
    Works with
    Docker
    Drop score
    85 of 100

    Full stat block for Langfuse

  2. Rank 46

    Opik

    Evals

    Traces agent runs and LLM calls, scores them with LLM-as-a-judge metrics and monitors production

    • score 53
    • 22.6k stars
    • pushed
    • 2.2.100 on
    • Apache-2.0

    Install the Python SDK

    pip install opik
opik configure
    Use it when, skip it if, stats for Opik

    Why you'd want it

    Opik, built by Comet, logs nested traces through a @track decorator in Python, a TypeScript SDK or OpenTelemetry, so multi-step agent runs and tool calls show up as full trace trees. It adds datasets, experiments, LLM-as-a-judge metrics for hallucination, moderation and RAG, a PyTest integration for CI, online evaluation rules, dashboards and an Agent Optimizer SDK. The full platform, backend included, is Apache-2.0 and can be self-hosted with Docker Compose or Helm, or used on Comet's cloud.

    Best for

    • Teams that want a fully Apache-2.0 self-hosted platform
    • Agent tracing and evals in one tool
    • Running LLM evals in PyTest

    Use it when

    You want tracing, evaluation and production monitoring under one open source license you can self-host, or want your coding agent to read traces through Opik's MCP server.

    Skip it if

    You only need lightweight offline tests; self-hosting runs a multi-container stack with databases and caches, and the alternative is a Comet account.

    Stats

    Stars
    22,610
    Last push
    Release
    2.2.100,
    License
    Apache-2.0
    Pricing
    Open source
    Works with
    Linux, macOS, Windows, Docker
    Drop score
    53 of 100

    Full stat block for Opik

  3. Rank 24

    Arize Phoenix

    Evals

    Traces LLM and agent apps over OpenTelemetry and runs evals, datasets and experiments in a local UI

    • score 63
    • 11.8k stars
    • pushed
    • arize-phoenix-v20.21.4 on
    • Elastic License 2.0

    Install with pip

    pip install arize-phoenix
phoenix serve
    Use it when, skip it if, stats for Arize Phoenix

    Why you'd want it

    Phoenix collects traces through OpenTelemetry-based OpenInference instrumentation, with integrations for frameworks such as the OpenAI Agents SDK, Claude Agent SDK, LangGraph and Vercel AI SDK across Python, JavaScript, Java and Go. On top of traces it offers LLM-based evals, versioned datasets, experiments, a prompt playground and prompt management. It runs locally with phoenix serve, in a container or on Kubernetes, and exposes a remote MCP endpoint so Claude Code, Cursor and other MCP clients can query traces, datasets and experiments.

    Best for

    • Local trace debugging during development
    • OpenTelemetry-based stacks
    • Self-hosted eval and experiment tracking

    Use it when

    You want an observability and eval UI on your own machine or infrastructure, instrumented with OpenTelemetry rather than a vendor-specific SDK.

    Skip it if

    You need an OSI open source license or want to offer it as a hosted service; the Elastic License 2.0 forbids that. For a managed platform Arize points to its separate Arize AX product.

    Stats

    Stars
    11,839
    Last push
    Release
    arize-phoenix-v20.21.4,
    License
    Elastic License 2.0
    Pricing
    Free
    Works with
    Docker
    Drop score
    63 of 100

    Full stat block for Arize Phoenix

For agents

The same entry, as files an agent can read without scraping this page. 

# read one entry
curl -s https://agenttooldrop.com/tools/langsmith.md

# list the MCP servers in the catalog
curl -s https://agenttooldrop.com/tools.json \
  | jq -r '.tools[] | select(.category=="mcp-server") | .slug'