Hugging Face, AWS partner in open-source machine learning amid AI arms race

Impressive advances in large language models (LLMs) are showing signs of what could be the start of a major shift in the technology industry. AI startups and big tech companies are finding new ways to leverage advanced LLMs for everything from composing emails to generating software code.

However, the promises of LLMs have also led to an arms race between tech giants. In their efforts to build their AI arsenals, big tech companies threaten to push the field toward less openness and more secrecy.

In the midst of this rivalry, Cuddle Sight charts another strategy that will provide scalable access to open-source AI models. Hugging Face partners with Amazon Web Services (AWS) to facilitate the adoption of open-source machine learning (ML) models. In an era where advanced models are increasingly inaccessible or hidden behind walled gardens, an easy-to-use open-source alternative can expand the market for applied machine learning.

Open source models

While large-scale machine learning models are very useful, setting them up and running them requires special expertise that few companies possess. The new partnership between Hugging Face and AWS will attempt to address these challenges.

Developers can use Amazon’s cloud tools and infrastructure to easily refine and deploy advanced models from Hugging Face’s ML repository.

The two companies began working in 2021 with the release of Hugging Face deep learning containers (DLCs) on SageMaker, Amazon’s cloud-based machine learning platform. The new partnership will extend the availability of Hugging Face models to other AWS products and Amazon’s cloud-based AI accelerator hardware to accelerate training and inference.

“Since we started offering Hugging Face natively in SageMaker, usage has grown exponentially and we now have over 1,000 customers using our solutions every month,” Jeff Boudier, product director at Hugging Face, told VentureBeat. “Through this new partnership, we are now working hand-in-hand with the engineering teams building new efficient hardware for AI, such as AWS Trainium and AWS Inferentia, to build solutions that are ready to run on Elastic Compute Cloud (EC2) and Elastic Kubernetes. Service (EKS).”

The AI ​​arms race

Tech leaders have been talking about the transformative nature of machine learning for several years now. But never has this transformation been felt as it has been in recent months. The release of OpenAI’s ChatGPT language model has set the stage for a new chapter in the race for AI dominance.

Microsoft recently poured $10 billion into OpenAI and is working hard to integrate LLMs into its products. Google has invested $300 million in Anthropic, a rival to OpenAI, and is doing everything it can to protect its online search empire from the rise of LLM-powered products.

These partnerships have clear benefits. With Microsoft’s financial backing, OpenAI has been able to train very large and expensive machine learning models on specialized hardware and deploy them at scale to millions of people. Anthropic will also gain special access to the Google Cloud Platform through its new partnership.

However, the rivalry between major tech companies also has drawbacks for the field. For example, since it began working with Microsoft, OpenAI has stopped open sourcing most of its machine learning models and serves them through a paid application programming interface (API). It has also become locked into Microsoft’s cloud platform and its models are only available on Azure and Microsoft products.

On the other hand, Hugging Face remains committed to continuing to provide open-source models. The collaboration between Hugging Face and Amazon allows developers and researchers to deploy open source models such as BLOOMZ (a GPT-3 alternative) and Stable Diffusion (a rival to DALL-E 2).

“This is an alliance between the open source machine learning leader and the cloud services leader to jointly build the next generation of open source models and solutions to use them. Everything we build together will be open-source and openly accessible,” said Boudier.

Hugging Face also wants to avoid the kind of lock-in that other AI companies face. While Amazon remains the cloud provider of choice, Hugging Face continues to partner with other cloud platforms.

“This new partnership is not exclusive and does not change our relationships with other cloud providers,” said Boudier. “Our mission is to democratize good machine learning, and to do that we need to empower users everywhere they use our models and libraries. We will continue to work with Microsoft and other clouds to serve customers everywhere.”

Openness and transparency

OpenAI’s API model is a useful option for companies that lack in-house ML expertise. Hugging Face also provides a similar service through its Inference Endpoint and Inference API products. But APIs will prove limited for organizations that want more flexibility to customize and integrate the models with other machine learning architectures. They are also inconvenient for research that requires access to model weights, inclines, and training data.

Easy-to-deploy, scalable cloud tools, such as those from Hugging Face, make these types of applications possible. At the same time, the company is developing tools to detect and flag abuse, bias, and other issues with ML models.

“Our vision is that openness and transparency [are] the way forward for ML,” said Boudier. “ML is science-driven and science requires reproducibility. Ease of use makes everything accessible to the end users so people can understand what models can and cannot do, [and] how they should and should not be used.”