When developing LLM applications, we consider performance issues during LLM calls and monitor outputs during the process.

At this point, tools like LangSmith and Langfuse become very useful.

However, sometimes we have local computing resources and prefer not to use cloud-based resources for LLM call monitoring, so we might not consider LangSmith.

In such cases, we can use Langfuse for this purpose.

Deployment

Deploying Langfuse is very simple; all you need to do is:

1
2
3

git clone https://github.com/langfuse/langfuse.git
cd langfuse
docker compose up -d

This way, the deployment is successful.

Replacement

If you previously used OpenAI’s SDK, you can continue using it as follows.

Install langfuse in the project:

1

pip install langfuse

To configure the API key, you need to use it in the deployed langfuse:

1
2
3

LANGFUSE_SECRET_KEY=<secret key>
LANGFUSE_PUBLIC_KEY=<public key>
LANGFUSE_HOST="http://localhost:3001"

Here I have set the Langfuse port to 3001; you should adjust according to your own configuration.

Simply replace the original OpenAI configuration:

1
2
3

# remove: import openai

from langfuse.openai import openai

In addition, langfuse also supports langchain and llamaindex, which will not be elaborated on further here.

Thoughts

Coze is also developing a large model agent framework, but the approach is quite different. Coze is building everything, including workflows and LLMs, making it relatively closed.

However, langfuse is more open, allowing the use of langchain and other models.

As a developer from a small company, I prefer the langfuse model because it offers more choices. However, if the project timeline is tight and Coze is barely usable, I would choose Coze.

Issues

1. An exception occurred when I replaced the OpenAI SDK:

1

Unexpected error occurred. Please check your request and contact support: https://langfuse.com/support.

2. I still encountered issues when testing test_langfuse.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

import os

from langfuse.decorators import observe
from langfuse.openai import openai


@observe()
def story():
  return (
    openai.chat.completions.create(
      model="moonshot-v1-auto",
      max_tokens=100,
      messages=[
        {"role": "system", "content": "You are a great storyteller."},
        {"role": "user", "content": "Once upon a time in a galaxy far, far away..."},
      ],
    )
    .choices[0]
    .message.content
  )


@observe()
def main():
  return story()


def test_langfuse():
  assert os.getenv("OPENAI_BASE_URL") is not None
  assert os.getenv("OPENAI_API_KEY") is not None
  main()

Regarding this issue, I have opened a discussion.

Additionally, if you wish to view the original code, you can obtain it from https://github.com/svtter/pdf-reader.