A leading engineering conglomerate sought a solution to deploy a powerful ChatBot on-premises, ensuring data privacy and security. In collaboration with ConSol, a project was initiated, envisioning the implementation of a ChatBot on an on-premises OpenShift Container Platform with NVIDIA GPUs, allowing for simultaneous use of multiple Large Language Models (LLMs) such as "Llama2-Chat" and "CodeLlama".
Project Description
The engineering conglomerate aimed to deploy a ChatBot offering similar functionality to ChatGPT, requiring no internet connection to ensure data privacy and security. The project involved deployment on an on-premises OpenShift Container Platform with NVIDIA GPUs and simultaneous use of various Large Language Models (LLMs) like "Llama2-Chat" and "CodeLlama".
Project Targets
The primary objective was to provide a secure and powerful ChatBot solution compliant with data privacy regulations and capable of efficiently handling complex queries. By implementing on an on-premises platform with NVIDIA GPUs, fast and accurate processing of natural language and high-quality responses were enabled.
Challenges included selecting open-source software to operate the models, determining the optimal model size for the available GPU memory, and efficiently utilizing the GPUs. The ChatBot had to be deployed and loaded offline without requiring an internet connection. Additionally, container images were transitioned to Red Hat's Universal Base Image (UBI) to mitigate container vulnerabilities.
ConSol handled the complete deployment of the ChatBot application and enhanced security by blocking internet access and transitioning to Red Hat UBI images. Standard processes such as CI/CD pipelines and documentation were adhered to ensure smooth implementation.
Employees of the engineering conglomerate benefit from a secure alternative to ChatGPT or similar solutions. The platform allows execution of open-source models, including newer ones such as "Mixtral 8x7B." By experimenting with local LLMs, including fine-tuning with proprietary data, the conglomerate can now build expertise and operate the optimal model for the desired use case.
Founded in
1984
More than
260
employees
In
4
countries
100%
creativity
50% of our staff have been at the company
10
years or more
More than
999
successful projects
100%
owner-managed
5
company venues
Contact
Samuel Pabst