How to Build an Enterprise Deep Learning Platform, Part Three

Check out an example deep learning platform in action in this Jupyter Notebook!

In parts one and two of this series we discussed why you need advanced tools to scale and accelerate your deep learning team, and showed you which open source tools will help fill those gaps. In this post we’ll stitch together three open source tools into a platform that enables end-to-end deep learning development.

As introduced in the previous parts, we’ll use a collection of open source tools for our sample platform: Pachyderm for data management, Determined for model development, and Seldon Core for deployment. We’ll install Pachyderm and Seldon on an EKS (kubernetes) cluster backed by S3. We’ll configure Determined to use our AWS account to automatically scale GPU instances to meet our team’s demand.

alt text

To see the platform in action, check out this Jupyter notebook example, which walks through:

  • Creating data repositories with Pachyderm and using Pachyderm data pipelines to prepare the data for training

  • Building a model within the Determined platform, and using its Adaptive hyperparameter search to quickly find a high-performing model

  • Building and testing a Seldon Core endpoint using the trained model produced by Determined.

This simple collection of tools (configurable in less than a day, and entirely open source) will greatly ease your ML team’s scaling challenges. They’ll be able to use this platform to generate more, higher-performing models and to quickly scale them to production to generate business value for your company.

If you have any questions about how you can use Determined as a part of your machine learning platform, join our community Slack or feel free to reach out to me directly!

Recent Posts

AUG 05, 2020

YogaDL: a better approach to data loading for deep learning models

JUL 30, 2020

TensorFlow Datasets: The Bad Parts

JUL 30, 2020

Choosing Your Deep Learning Infrastructure: The Cloud vs. On-Prem Debate

Stay updated