TensTorrent and Cerebras - Novel Advanced Computer Architectures for Deep Learning/Neural Networks Training

Interesting systems to check:

TensTorrent Startup: https://www.tenstorrent.com/technology/

Their architecture offers more independent parallelism of the cores than the current GPUs/TPUs, or as they express it: "Fully programmable architecture that supports fine-grain conditional execution, dynamic sparsity handling, and an unprecedented ability to scale via tight integration of computation and networking"

Cerebras Systems: Building the World’s First Wafer-Scale Processor

https://youtu.be/LiJaHflemKU

An amazing hugely-multi-core processor with 1.2 Trillion/2.6 Trillion transistors, with defect redundancy (the system can tolerate the presence of some defected cores which are expected) and inter-chip buses, which covers an entire silicon wafer. Their first version CS1 claims 400000 "AI-Optimized" cores, while the second CS2: 850000 cores, also 18 GB and 40 GB respectively on-chip memory, accessible in one clock. The die size is 46225 sq.mm compared to 815 sq. mm and 21.1 Billion transistors for a given biggest-die GPU at the time (2020).

How The World's Largest AI/ML Training System Was Built (Cerebras)

6857 показвания •18.07.2020 г. NASJAQ

https://youtu.be/Fcob512SJz0


https://cerebras.net/product/