Agnostiq & MongoDB: High-Performance Computing for All

Material scientists, computational biologists, and AI researchers all have at least one thing in common; the need for huge amounts of processing power, or 'compute', to turn raw data into results.

But here's the problem. Many researchers lack the skills needed to build the workflows that move data through huge networks of distributed servers, CPUs, and GPUs that actually do the number crunching.

And that's where Agnostiq comes in. Since the company's inception in 2018, Agnostiq has put the power of high-performance computing (HPC) in the hands of researchers, bypassing the need for development expertise to build these essential data and compute pipelines.

Power to the people

"We started research on high-performance computing needs in fields like finance and chemistry, and through the process of onboarding, researchers quickly realized how hard it was for [researchers] to access and scale up on the cloud, or tap into HPC and GPU resources,'' said Santosh Kumar Radha, Agnostiq's Head of Product. "If you wanted to scale up, there were not many tools available in the major cloud providers to do this."

To address this bottleneck, the team at Agnostiq built Covalent, a Python-based framework that allows researchers to easily design and run massive compute jobs on cloud platforms, on-prem clusters, and HPC services. With Covalent, startups and enterprises can build any AI or HPC application in a simple, scalable, and cost-effective way using a Python notebook, negating the need to interact with underlying infrastructure.

One of the hardest challenges the Covalent team faced was combining traditional HPC with modern cloud technology. Because traditional HPC infrastructure was never designed to run in the cloud, the team spent considerable resources marrying techniques like GPU and CPU parallelization, task parallelization, and graph optimization with distributed cloud computing environments.

As a result, researchers can use Covalent to quickly create a workflow that combines the convenience of cloud computing with specialized GPU providers and other HPC services.

Everything, everywhere, all at once

As the name suggests, Agnostiq has always focused on making their platform as open and resource neutral as possible.

MongoDB Atlas, with its native multi-cloud capability, was a perfect complement.

"At Agnostiq, everything we build has to be technology and vendor neutral. Interoperability is key for us," said Radha. "We do all the mapping for our customers, so our platform has to perform a seamless transition from cloud to cloud."

The ability to move data between clouds became even more critical following the release of ChatGPT. With an explosion in generative AI research and development, the availability of GPU resources plummeted.

"Resource scarcity in the 'GPT era' means you couldn't get access to GPUs anywhere," Radha added. "If you didn't have a default cloud posture, you were nowhere, which is why we doubled down on multi-cloud and MongoDB Atlas to give our clients that optionality."

Open source opening doors

Since the beginning, the team at Agnostiq has chosen MongoDB as their default NoSQL database.

At first, the team adopted MongoDB's free, open source product. "We didn't have any DBAs as a small agile team. MongoDB gave us the freedom to build and manage our data workflows without the need for a specialist," said William Cunningham, Head of HPC at Agnostiq.

As their customer base grew along with the demand for cloud computing access, Agnostiq moved to MongoDB Atlas, gaining the freedom to move data seamlessly between AWS, Google Cloud, and Microsoft Azure. This gave Covalent the flexibility to reach multi-cloud compatibility at a faster rate than with standard tooling.

Covalent provides a workflow management service by registering jobs, dispatching IDs, and collecting other metadata that allows fellow researchers and developers to reproduce the original work. MongoDB is used in the front-end, allowing a high volume of metadata and other assets to be published and cached in accordance with an event-driven architecture. This near real-time experience is key to a product aimed at delivering a unified view over distributed resources. MongoDB Atlas further provided the autoscaling required to grow with the user base and the number of workloads while keeping costs in check.

"MongoDB Atlas helps us provide an ideal foundation for modern HPC and AI applications which require serverless compute, autoscaling resources, distributed workloads, and rapidly reconfigurable infrastructure," added Radha.

The future

Looking to the future, Agnostiq is focused on servicing the huge demand for gen AI modeling and workflow building. To that end, the company released its own inference service called Function Serve within Covalent. Function Serve offers customers a complete, enterprise-grade solution for AI development and deployment, supporting serverless AI model training and fine-tuning.

With Function Serve, customers can fine-tune, host, and serve any open-source or proprietary model with full infrastructure abstraction, all with only a few additional lines of code. MongoDB Atlas was used to rapidly develop a minimal service catalog while remaining cloud-agnostic.

Looking ahead, the team plans to leverage MongoDB Atlas for enterprise and hybrid-cloud deployments in order to quickly meet customers in their existing cloud platforms.

Agnostiq is a member of the MongoDB AI Innovators program, providing their team with access to Atlas credits and technical best practices. You can get started with your AI-powered apps by registering for MongoDB Atlas and exploring the tutorials available in our AI resources center. Additionally, if your company is interested in being featured, we'd love to hear from you. Reach out to us at [email protected].

Public link

Please use the above public link if you want to share this noodl on another website.