Update on Determined AI and Pachyderm Open Source Community Efforts
March 20, 2024
Determined AI
Determined AI is not just another AI/ML platform. It stands out as a unique open source platform for ML at scale that is committed to fostering a collaborative and inclusive environment. With a team of dedicated machine learning developer advocates, a community manager, and a marketing/events coordinator, Determined AI fully supports the growing needs of its open source community.
At the core of Determined AI’s mission is the belief in the power of open source AI. By providing an open platform, Determined AI empowers developers and researchers to freely access and contribute to cutting-edge AI and machine learning technologies. This commitment to openness fosters innovation, accelerates progress, and enables a diverse community of contributors to come together and advance the field collectively.
The team at Determined AI comprises a group of machine learning, distributed training, systems, and user experience experts who bring a wealth of knowledge and experience to the table. With a deep understanding of AI and machine learning, we ensure that the platform remains robust, efficient, and continuously evolving. Our knowledge and experience allows us to address user challenges, provide technical support, and guide community members towards achieving their goals.
The open source community team aims to provide a rich and engaging experience for community members to achieve those goals. We organize a variety of informative events, including workshops, webinars, and hackathons, where community members can learn new skills, share ideas, and collaborate on projects. We contribute to documentation. The team also understands the importance of support, which is why we offer a dedicated Slack channel where community members can seek assistance, ask questions, and connect with like-minded individuals. We produce a newsletter that contains the latest news on AI and ML and updates on Determined features. Additionally, the team offers free personalized 1:1 sessions for community members to offer guidance and mentorship.
The open source community team also actively participates in AI/ML conferences, both as attendees and presenters, showcasing the platform’s capabilities and sharing insights with the wider AI and machine learning community. By providing these diverse avenues of engagement and support, the open source community team ensures that community members have access to resources, knowledge, and opportunities for growth.
HPE
HPE’s acquisition of Determined AI did not alter our unwavering commitment to open source. On the contrary, HPE recognizes the immense value of open source in driving AI innovation and remains dedicated to fostering a collaborative ecosystem. HPE understands that open source is instrumental in advancing AI technologies and enabling a diverse community to contribute to its development. The acquisition of Determined AI has provided us with even more resources to promote and support truly open source AI. With HPE’s backing, Determined AI is empowered to expand its reach, enhance its platform, and further advocate for the importance of open source in our AI strategy.
In addition to backing Determined AI, HPE offers a range of valuable resources and solutions to the AI and ML community. HPE understands the unique hardware requirements for ML at scale and provides advanced hardware solutions tailored to meet those needs. With their expertise in designing and delivering high-performance computing infrastructure, HPE empowers organizations to effectively leverage machine learning capabilities. Furthermore, HPE offers turnkey ML solutions, providing pre-configured and optimized systems that enable quick deployment and efficient utilization of Determined AI. For those seeking managed solutions, HPE offers comprehensive AI services, including end-to-end support, from infrastructure setup to model deployment and management. HPE’s commitment to the AI and ML community is evident in their dedication to providing not just technologies, but also the necessary support and expertise to drive successful AI initiatives.
Together, HPE and Determined AI are poised to accelerate the growth of open source AI and pave the way for transformative advancements in the field.
Pachyderm
Determined AI’s platform plays a crucial role in the ML workflow by enabling efficient and scalable model training. However, as HPE recognized the importance of a complete end-to-end ML workflow, we identified the need for robust data management capabilities as the foundation for successful ML projects. To address this, HPE made the strategic decision to acquire Pachyderm in 2023. By integrating Pachyderm into our open source ML stack, we aim to offer a complete and streamlined ML experience, starting from data collection and preparation, all the way to model training and deployment. This acquisition reinforces our commitment to open source and empowering the community and organizations with the necessary tools and technologies for effective and efficient ML workflows.
Pachyderm brings numerous benefits to ML training, making it an invaluable addition to our open source ML stack. Pachyderm offers robust data versioning and lineage capabilities, ensuring data integrity and reproducibility throughout the training process. This allows teams to easily track and manage changes to their datasets, facilitating collaboration and experimentation. Pachyderm provides scalable and distributed data processing, enabling efficient handling of large datasets and parallelizing data transformations. This results in faster and more efficient model training. Additionally, Pachyderm’s integration simplifies the deployment and orchestration of ML pipelines, reducing the complexity of managing the entire ML workflow. Pachyderm enhances ML training by Determined AI by providing comprehensive data management, scalability, reproducibility, and seamless integration our open source ML stack.
Looking Forward
With the acquisition of Pachyderm, the Determined AI open source community team broadened its mandate to encompass the entire open source ML stack, including both Determined AI and Pachyderm. The combined HPE AI Open Source Community team is dedicated to supporting the Pachyderm and Determined open source communities, fostering collaboration, knowledge sharing, and innovation. By bringing together these two powerful platforms, the team aims to provide ML practitioners with a holistic and integrated solution for their data management, model training, and deployment needs. With a shared commitment to open source principles, the HPE AI Open Source Community team will actively engage with the community, provide resources, and drive the advancement of open source ML technologies.
We are thrilled to welcome Pachyderm open source users to our newly combined open source community. For existing Pachyderm users, Determined AI offers a powerful set of tools and capabilities that enhance the model training and deployment aspects of the ML workflow. Determined is a comprehensive ML training platform that provides advanced features such as hyperparameter tuning, distributed training, and experiment tracking. By integrating Determined into their workflow, Pachyderm users can benefit from streamlined and efficient model training, enabling them to experiment with different configurations and optimize their models more effectively. Additionally, Determined offers seamless model deployment options, making it easier for Pachyderm users to take their trained models into production. With Determined AI, Pachyderm users can leverage a complete open source ML stack that covers both data management and model training, enabling them to accelerate their ML projects and achieve better results.
For existing Determined AI users, Pachyderm offers a robust and scalable data versioning and lineage solution that complements the model training capabilities of Determined. With Pachyderm, Determined users can efficiently manage and track changes to their data, ensuring reproducibility and traceability throughout the ML pipeline. Pachyderm’s data versioning capabilities enable users to easily access and revert to previous versions of their data, facilitating experimentation and iteration. Furthermore, Pachyderm’s scalable data processing capabilities allow users to handle large datasets and perform complex data transformations seamlessly. By integrating Pachyderm into their workflow, Determined AI users can enhance their data management practices, improve collaboration, and ensure the integrity of their ML projects from data ingestion to model deployment.
The HPE Open Source Community team is committed to fostering collaboration and knowledge sharing among our Determined AI and Pachyderm community members, and we look forward to exploring the synergies between Pachyderm and Determined, ultimately driving innovation and advancing the field of open source ML. We look forward to seeing you at our events, on the Slack, meeting you 1:1, and hearing your feedback.
Isha
Kevin
Alex
Hayden