A National Parks recommender using langchain4j

I have been playing around with langchain4j recently, to compare it against the capabilities of langchain implementation in Python.

These are quick thoughts, based on a week of trying it out:

1. It is great that a Java-based implementation that makes it possible for Java applications to use LLM, is available and is being actively developed
2. It still has some ways to go before it has parity with Python-based langchain though.

Test Application

To put the library through its paces, I wanted to write a simple application that acts as a recommender tool for US National parks. It should be able to hold a conversation which means that it should be able to hold the previous questions and responses in memory.

Crater Lake National Park

First attempt

So for the first attempt, I wanted to replicate the chain I had put together using langchain for a streamlit application. The chain looks something like this:

template = """Answer the question based on the context below and use the history of the conversation to continue
            If the question cannot be answered using the information provided answer with "I don't know"
                          
            Context:
            You are deeply knowledgeable about all the National Parks in the United states and you want to 
            suggest the parks to visit based on the question.
                          
            History of the conversation:
            {chat_history}
            
            Question: 
            {question}

            Answer: 
"""
prompt_template = PromptTemplate.from_template(template)
# Notice that we need to align the `memory_key`
memory = ConversationBufferMemory(memory_key="chat_history")
chain = LLMChain(
    llm=get_llm(),
    prompt=prompt_template,
    verbose=True,
    memory=memory
)

This works beautifully, the context sets up the agent as being knowledgable about US National Parks and a sample conversation using streamlit UI looks like this:

I assumed that a similar chain should work for langchain4j, however, it was not so simple.

There are two chains that langchain4j supports today:

1. ConversationalChain is a chain that enables a conversation with a model while maintaining history
2. ConversationalRetrievalChain is a chain that enables conversation using content stored in a retrieval system like a vector database. If say you want a conversation based on the content of a website or personal documents then this is the chain to use.

In my case ConversationalChain is appropriate. The first problem that I ran into is that the ConversationalChain does not surprisingly take in a prompt template! While it is easy to create a prompt template that looks a little different from the Python version:

import dev.langchain4j.model.input.PromptTemplate;

import java.util.List;
import java.util.Map;

PromptTemplate promptTemplate = PromptTemplate.from("""
        Answer the question based on the context below and use the history of the conversation to continue
        If the question cannot be answered using the information provided answer with "I don't know"
                      
        Context:
        You are deeply knowledgeable about all the National Parks in the United states and you want to 
        suggest the parks to visit based on the question.
                      
        History of the conversation:
        {{history}}
                        
        Question: 
        {{question}}

        Answer: 
        """);

promptTemplate.apply(Map.of(
        "question", 
        "What are the places to visit near my home state of Oregon?", 
        "history", List.of())).text()

There is no simple way to pass this prompt on to the ConversationalChain !

Second attempt

I realize it will be a matter of time before it is fixed. In the meantime, it is fairly easy to add in a chain that takes in a prompt template, my implementation looks like this —

package org.bk.langchain4j.service;

import dev.langchain4j.chain.Chain;
import dev.langchain4j.data.message.AiMessage;
import dev.langchain4j.data.message.UserMessage;
import dev.langchain4j.memory.ChatMemory;
import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.input.PromptTemplate;

import java.util.Map;
import java.util.stream.Collectors;

public class CustomConversationalChain implements Chain<String, String> {
    private static final Logger LOGGER = LoggerFactory.getLogger(CustomConversationalChain.class);

    private static final String MEMORY_KEY = "chat_history";
    private static final String QUESTION_KEY = "question";

    private final ChatLanguageModel chatLanguageModel;
    private final PromptTemplate promptTemplate;
    private final ChatMemory chatMemory;
    private final String memoryKey;

    ....

    @Override
    public String execute(String message) {
        Map<String, Object> withMemory = createMapWithMemory(chatMemory, message);
        String prompt = this.promptTemplate.apply(withMemory).text();
        LOGGER.debug("Generated prompt: {}", prompt);
        String aiResponse = chatLanguageModel.generate(prompt);
        chatMemory.add(UserMessage.userMessage(message));
        chatMemory.add(AiMessage.aiMessage(aiResponse));
        return aiResponse;

    }

    private Map<String, Object> createMapWithMemory(ChatMemory chatMemory, String userMessage) {
        return  Map.of(memoryKey, chatMemory.messages()
                        .stream()
                        .map(chatMessage -> chatMessage.type().name() + ":" + chatMessage.text())
                        .collect(Collectors.joining("\n")),
                QUESTION_KEY, userMessage);
    }

    ...
}

The core of this implementation is in using the prompt template to provide an appropriate context to the LLM and using that to continue the conversation, the prompt after a few conversations looks like this:

Answer the question based on the context below and use the history of the conversation to continue
If the question cannot be answered using the information provided answer with "I don't know"

Context:
You are deeply knowledgeable about all the National Parks in the United states and you want to
suggest the parks to visit based on the question.

History of the conversation:
USER:I stay in Oregon, any interesting places to visit around
AI:Yes, there are many beautiful National Parks near Oregon that you can visit. Some popular ones include Crater Lake National Park, Mount Rainier National Park, Olympic National Park, and Redwood National and State Parks. These parks offer stunning landscapes, hiking trails, and unique wildlife experiences.
USER:anything north of here
AI:Yes, there are several national parks located north of Oregon that you might find interesting to visit. One notable park is North Cascades National Park in Washington. It offers breathtaking mountain scenery, pristine lakes, and numerous hiking trails. Another option is Glacier National Park in Montana, known for its stunning glaciers, alpine meadows, and diverse wildlife. Both parks provide excellent opportunities for outdoor exploration and adventure.

Question:
anything south of here

Answer: 

Conclusion

I will be following(and hopefully contributing to) the langchain4j project with a great deal of interest as it builds parity with langchain. It will open up integrations with LLM in Java-based enterprise shops.

Here is my full Java project that uses langchain4j — https://github.com/bijukunjummen/langchain4j-demo

Here is the streamlit/langchain based implementation — https://github.com/bijukunjummen/genai-chat