Deployment of a powerful on-premises ChatBot

A leading engineering conglomerate sought a solution to deploy a powerful ChatBot on-premises, ensuring data privacy and security. In collaboration with ConSol, a project was initiated, envisioning the implementation of a ChatBot on an on-premises OpenShift Container Platform with NVIDIA GPUs, allowing for simultaneous use of multiple Large Language Models (LLMs) such as "Llama2-Chat" and "CodeLlama".

Project Description

The engineering conglomerate aimed to deploy a ChatBot offering similar functionality to ChatGPT, requiring no internet connection to ensure data privacy and security. The project involved deployment on an on-premises OpenShift Container Platform with NVIDIA GPUs and simultaneous use of various Large Language Models (LLMs) like "Llama2-Chat" and "CodeLlama".

 

Project Targets

The primary objective was to provide a secure and powerful ChatBot solution compliant with data privacy regulations and capable of efficiently handling complex queries. By implementing on an on-premises platform with NVIDIA GPUs, fast and accurate processing of natural language and high-quality responses were enabled.

 

Challenges included selecting open-source software to operate the models, determining the optimal model size for the available GPU memory, and efficiently utilizing the GPUs. The ChatBot had to be deployed and loaded offline without requiring an internet connection. Additionally, container images were transitioned to Red Hat's Universal Base Image (UBI) to mitigate container vulnerabilities.

Challenge

Solution

ConSol handled the complete deployment of the ChatBot application and enhanced security by blocking internet access and transitioning to Red Hat UBI images. Standard processes such as CI/CD pipelines and documentation were adhered to ensure smooth implementation.

Employees of the engineering conglomerate benefit from a secure alternative to ChatGPT or similar solutions. The platform allows execution of open-source models, including newer ones such as "Mixtral 8x7B." By experimenting with local LLMs, including fine-tuning with proprietary data, the conglomerate can now build expertise and operate the optimal model for the desired use case.

Benefits

Founded in

1984

 

More than

260

employees

 

In

4

countries

 

100%

creativity

50% of our staff have been at the company

10

years or more

More than

999

successful projects

100%

owner-managed

 

5

company venues

 

Do you have any further questions about our portfolio?

Let's talk!

Andreas Schilz

# Red Hat
# Cloud
# Platform Engineering
+49-211-339903-86
By submitting the form, you agree to our privacy policy.