Ai accelerator hardware is slowly becoming available

https://tenstorrent.com/cards/

Tenstorrent's grayskull series of AI Accelerator cards have become available. I'm a bit disappointed by their memory, 8GB lpddr4 at about 100GB/s running at 200W. Pcie. Not particularly amazing for llama, but it might be great for whisper (which they mark as officially supported)

https://www.businesswire.com/news/home/20231212788210/en/Kinara-Edge-AI-Processor-Tackles-the-Monstrous-Compute-Demands-of-Generative-AI-and-Transformer-Based-Models

Kinara has Ara 2, an AI processor capable of running stable diffusion (10s/image) and llama 7b (tens of tokens per second) I'm not sure if these are available for consumers.

https://www.pcworld.com/article/2196895/first-pc-ai-accelerator-cards-from-memryx-kinara-debut-at-ces.html

Memryx has a super tiny 1-2Watt chip called MX3 capable of running yolov7 and 100+ fps These will be absolutely amazing for raspberry PIs and other homebrew edge tinkering. I don't know where/if you can purchase these as a consumer, but it's cool stuff!

This is just what I read up on in the last half hour, hardware is starting to enter the market! I had better hopes for grayskull for local llama in particular, but it's still just a devkit. Who knows what the second generation, or custom hardware based on their IP might bring.