{"id":678044,"date":"2026-10-29T21:06:00","date_gmt":"2026-10-29T21:06:00","guid":{"rendered":"https:\/\/Blockchain.News\/news\/assemblyai-raw-websocket-voice-agent-no-sdk"},"modified":"2026-10-29T21:06:00","modified_gmt":"2026-10-29T21:06:00","slug":"assemblyai-details-raw-websocket-voice-agent-build-with-no-sdk","status":"publish","type":"post","link":"https:\/\/e-bitco.in\/index.php\/2026\/10\/29\/assemblyai-details-raw-websocket-voice-agent-build-with-no-sdk\/","title":{"rendered":"AssemblyAI details raw WebSocket voice agent build with no SDK"},"content":{"rendered":"<div id>\n<nav class=\"bn-article-breadcrumb\" aria-label=\"Breadcrumb\"> <a href=\"https:\/\/blockchain.news\/\">Home<\/a> <span aria-hidden=\"true\">\/<\/span> <a href=\"https:\/\/blockchain.news\/#home-latest-heading\">Latest reporting<\/a> <\/nav>\n<p class=\"bn-article-type\">News<\/p>\n<p class=\"lead\">AssemblyAI published a tutorial for building a voice agent directly on WebSockets with streaming STT, LLM Gateway and TTS.<\/p>\n<div class=\"bn-article-byline\"> <a href=\"https:\/\/blockchain.news\/Profile\/Terrill-Dicki\" rel=\"author\">Terrill Dicki<\/a> Oct 29, 2026 21:06 UTC <\/div>\n<figure class=\"bn-article-hero\"> <a href=\"https:\/\/blockchain.news\/bn\/images\/editorial\/markets-technology.jpg\" class=\"hero-image-link\"> <img fetchpriority=\"high\" decoding=\"async\" class=\"hero-image\" src=\"https:\/\/blockchain.news\/bn\/images\/editorial\/markets-technology.jpg\" alt=\"Thematic editorial illustration\" loading=\"eager\" width=\"1200\" height=\"675\"> <\/a><figcaption>Editorial illustration; not a product screenshot or event photograph<\/figcaption><\/figure>\n<\/div>\n<div data-article-body>\n<p>AssemblyAI published a tutorial on Oct. 29, 2026 that walks through building a voice agent directly on raw WebSockets, without using its SDK. The post ties together streaming speech-to-text, an HTTP LLM call and text-to-speech playback, and frames the approach as a way to keep full control over every protocol frame and state transition.<\/p>\n<h2>Evidence and context<\/h2>\n<p>The guide uses three connections and one API key: the Universal-3.6 Pro Realtime speech-to-text WebSocket, LLM Gateway over plain HTTPS, and the AssemblyAI Text-to-Speech WebSocket. According to the tutorial, the speech-to-text socket receives 16 kHz binary PCM audio and returns frames such as Begin, SpeechStarted, Turn and Termination, while the text-to-speech socket accepts Generate, Flush, Cancel, KeepAlive and Terminate and returns Audio, FlushDone, Cancelled, Termination and Error.<\/p>\n<p>AssemblyAI says the example is designed for developers who want to understand or own each byte of the flow, including barge-in behavior and tool calling. The post says the sample fits in about 300 lines of Python and uses only the websocket, httpx and sounddevice libraries. It also says the same API key authenticates all three services, with LLM Gateway using an OpenAI-compatible chat-completions format.<\/p>\n<p>The tutorial\u2019s example agent is configured to answer weather questions via a get_weather tool call. In the sample code, the model request is sent with tools in the message payload, the function result is appended as a tool message, and the LLM is called again until it returns plain text. AssemblyAI\u2019s sample then sends that text to the TTS socket, where each reply must be followed by Flush to produce audio.<\/p>\n<p>The article also documents barge-in handling: when the user speaks over the agent, the program clears local playback and sends Cancel on the TTS socket, while dropping any trailing Audio frames until Cancelled arrives. It adds a backchannel filter so short acknowledgements such as \u201cmhm,\u201d \u201cokay\u201d or \u201cyeah\u201d do not interrupt the agent mid-sentence. The tutorial says the TTS session can stay open for the whole conversation, but should be kept alive with periodic frames during idle periods.<\/p>\n<p>AssemblyAI\u2019s post includes operational details and error handling guidance, including the note that the TTS socket sends an Error frame before failure closes and that the STT socket closes with websocket close codes and reasons. The guide also says the TTS voice in the sample uses Audrey\u2019s voice ID, that the output format is 24 kHz pcm_s16le, and that the microphone input should match the 16 kHz sample rate configured on the speech-to-text side.<\/p>\n<\/p><\/div>\n<p><a href=\"https:\/\/blockchain.news\/\">Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Home \/ Latest reporting News AssemblyAI published a tutorial for building a voice agent directly on WebSockets with streaming STT, LLM Gateway and TTS. Terrill Dicki Oct 29, 2026 21:06 UTC Editorial illustration; not a product screenshot or event photograph AssemblyAI published a tutorial on Oct. 29, 2026 that walks through building a voice agent [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":678045,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[19529,25,1103,20554,19856,24950],"class_list":{"0":"post-678044","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-blockchain","8":"tag-assemblyai","9":"tag-news","10":"tag-streaming","11":"tag-tool-calling","12":"tag-voice-ai","13":"tag-websockets"},"_links":{"self":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/posts\/678044","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/comments?post=678044"}],"version-history":[{"count":0,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/posts\/678044\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/media\/678045"}],"wp:attachment":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/media?parent=678044"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/categories?post=678044"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/tags?post=678044"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}