• GPUs are big and heavy! And expensive….
• The ability of having server which can generate tokens “for free” I would say is very libereting. You are just testing and running things against your own server and you are not worrying about your bills.
• You can run some fantastic LLMs on 5 years old hardware (my GPU is RTX 3090). This is probably one of the most fascinating things I have found out this week. Today’s models which you can fit into 24GB VRAM are seriously good and on the level of LLMs we had to pay OpenAI or Google last year. My current LLMs which I’m testing is Qwen3-Coder-30B and the things it can build are crazy good.
• I have switched from using LLM as assistant in your VScode to running agents (semi)autonomously like Goose AI. Goose is currently my favourite agent. It is open source and you can run it either in GUI or in console. It supports tools, sub-tools and MCP so it can do lots of interesting stuff.
• Speaking of Goose I’m learning how to use agents more efficiently - workflows of to plan and implement with agent is something I’m starting to learn and I would say getting better in it. It is fascinating seeing agents doing their own thing. I’m learning how to build Skills and use receipts.
• I chose vLLM as my platform to run LLMs. It took little bit of time of testing but I think nailed it down so everything works as it should be. One weakness is probably limited GGUF support.
• The pace is high. vLLM has new release every other week with new features. New better model is being dropped every other month on Huggingface.

Kubernetes I have installed pretty much the standard stack - Cillium, CSI NFS, Grafana, Prometheus. Everything is managed with GitOps using Flux. I have decided day 1 to deploy Vault for secrets management which can get handy in the future. The plan is to deploy vLLM on this stack and probably migrate my web server on the cluster. I have Jupyter playbooks running so it is ready for AI/ML sandboxing.

vLLM VM Speaking to vLLM this was completely new. Never tried to run vLLM. I have initially tried to run it on Kubernetes but with my limited memory and no GPU (yet) I have managed to kill my cluster few times. I have decided to take a step back and isolate vLLM from anything else which showed to be better approach. I’m now running small Qwen/Qwen3-1.7B model. It is more for testing purposes but pretty cool I can run small LLM on CPU only.

Goose I was starting to play with opensource AI agents. This is completely new space for me so was starting to testing it with Qwen model running vLLM. Does Goose support it? Yes. Is it useful with small Qwen model? Not really. I will have to wait to use with larger local models. I have tested it very briefly with Gemini 3.0 Pro and it worked great.

I have done lots of backend work outside of AI stack like Portworx backups, web server, mail server, Obsidian, important Grafana dashboards etc.

I think I have now solid platform I can start building on. In the following weeks I want to spend more time on tuning vLLM and probably migrate it to Kubernetes. Plan is to buy GPU in December to start more progressing on AI and finally start focusing on that.

Regarding AI specifically, the move from isolated chatbots towards agentic AI seems undeniable. If you consider that the market was already fragmented during the “cloud native” era a few years ago, it has become even more so in today’s AI age. New tools are being created almost weekly.

This presents a challenge for vendors like Cisco. Attempting to differentiate might focus on performance – but this isn’t likely the factor where they compete against giants like Nvidia. Instead, Cisco’s leadership appears to believe differentiation lies in AI security, which could be a valid proposition. However, Cisco should also consider adding more vertical integration capabilities to their offering and more on AI inferencing.

Let me explain why this might be an opportunity for Cisco. Traditionally, AI builders have relied on public clouds (like AWS or Azure) which provide easy deployment and operation, reducing initial hardware investment needs. These platforms also allow for rapid iteration during the development phase before potentially moving workloads to production. However, when an AI project succeeds and moves towards production deployment, cloud costs frequently get out of control. This forces infrastructure teams to explore “on-premises” alternatives to manage costs.

Moving AI workloads “on premises” isn’t simply about deploying a few server GPUs. It requires the full stack – data, models, inference environment – running on top of robust hardware layers. While public clouds excel here, building a comparable “on-prem” solution is still very challenging.

Focus on inferencing, however, will be crucial. It is great that Cisco is the only networking vendor who provides compatibility with Spectrum X from Nvidia because I understand that training large models is where the significant investment often happens , however companies – especially their enterprise clients – primarily run these models rather than training them and this will only be growing.

There were some interesting announcements around own Cisco models i.e. Deep Network Model. I’m planing to do little bit more testing on this model and see in what tasks will it exactly stand out against models like GPT5 etc. There will be new Time Series Foundational Model being released in November 2025. However, I like the idea that vendors will be in the future building their models which than should allow them building their own more specialised agents - I think.

Moving more into the long term vision Cisco seems determined to play a more active role in Agentic AI, potentially through its AGNTCY project. The same way they were central to defining infrastructure for the Internet in the 90s, perhaps their goal now is to shape “agentic infrastructure.” This involves standardizing the way agents interact, communicating with each other, thereby addressing complexities associated with tools like LangChain, LangGraph, and MCP.

I like how Cisco takes a wider approach and experiments more than some of their competitors with new concepts. The key is now to address underlaying challenges around their compute portfolio and focus more on AI software stack and inferencing, not just training.

Final specs

Without long overdue this is the final built:

Build

It was fairly smooth sail. I have put so much effort into planing, reading and researching that the most of the built was pretty much done in one afternoon. Open the chassis, install all 4 fans, put in motherboard risers, attach M.2s to the motherboard, put in motherboard, plug in power supply, put in harddrives and connect everything up. Done. It would be nice to just power it on and see what happens but I was missing memories due to long lead times. That didn’t stop me so I wanted to test if I can at least connect to BMC (Supermicro out of band)…And I could. So if you ever wonder would I be able to connect to BMC without memories the answer is yes.

It looks like lead times for server graded DDR5 memories are currently in weeks so after 2+ weeks of waiting I had my memories and was able to power it up. And it came up without any issues! Massive success.

Hypervisor

I have done lots of thinking during waiting for my memories what hypervisor I will go for. I was considering Ubuntu KVM and Proxmox and in the end went for Proxmox. I felt like hypervisor configuration and operation is something where I want to spend minimum time and Proxmox does that. It is running on Ubuntu and after playing little bit with it is offering flexibility of Ubuntu with some extra features like nice GUI. Definitely recommend it.

Running it

I have started on Friday night and after few hours on Saturday and Sunday I was able to configure hypervisor, install web server and basic monitoring so I could migrate my couple of AWS servers on my new home server including this blog. I have even managed to set up backups on my NAS.

What next?

I still have to migrate my personal mail server from AWS which means I can shutdown my whole AWS environment. I will work on design deploying Kubernetes cluster where I’m planing to eventually migrate my web server and make it ready for the future AI experiments. After I will have this I will start looking on things like rack and obviously GPU. I think the server (but mainly my wallet) will be ready for the GPU towards the end of October.

I think “agent” may finally have a widely enough agreed upon definition to be useful jargon now

The example the industry feels the same is Huggingface introducing new GAIA tests called (surprise) Gaia2.

Where GAIA was read-only, Gaia2 is now a read-and-write benchmark, focusing on interactive behavior and complexity management.

In their new framework they are approaching the whole benchmark as a test how human would be using agents to achieve their goals i.e. sending emails, create calendar events or simply chatting to other agents. I think this reflects the way we are starting to see using LLM: not just in “read-only” but in “read-write mode” where agents are becoming more and more interactive. Interesting document where Huggingface team describe how the benchamrk works, what they are testing and how you can run the test yourself. They are presenting their first results using typical models like Llama 3.3, GPT-4o or Gemini.

I was able to meet with Positron only briefly and few interesting facts about them:

• Co-founded by ex-Groq and ex-LambdaLabs, 30 employees
• They just went through series A funding
• Delivering their own AI inferencing product to traditional datacenters
• They are not focusing on speed - they focus on performance/W and performance/$
• They see future of inferencing on agentic MoE architecture - collection of small LLMs
• At this stage they are focusing on larger deployments
  • Offering AI server with 8x Positron Archer Transformer Accelerators, each with 32GB HBM

I will have a follow up with Positron focusing on their technical architecture in following few weeks so I will follow up with more detailed post. Very exciting to see new hardware vendors - this is exciting field opened to new companies.

It is fascinating to see where robotics managed to get in the last few years where things like LLM, LVM (Large Vision Model) and hardware engineering merging together. We are at the very exciting times when it comes to robotics.

Looking little bit more deeper on what should be the right architecture for AI workloads in my homelab I came across llm-d. It has been launched by CoreWeave, Google, IBM Research, NVIDIA, and Red Hat. Their statement really resonates:

The objective of llm-d is to create a well-lit path for anyone to adopt the leading distributed inference optimizations within their existing deployment framework - Kubernetes.

Llm-d building blocks are vLLM as inferencing engine, K8s as a core platform and Inference gateway to provide intelligent scheduling which is build for LLM type of workloads. I would highly recommend spend a bit more time to read through their announcement which explains very nicely the differences between typical workloads and LLM workloads.

Even though the focus is on deploying large scale inference on Kubernetes using large models (e.g. Llama-70B+, not Llama-8B) with longer input sequence lengths (e.g 10k ISL | 1k OSL, not 200 ISL | 200 OSL) and mostly tested on 16 or 8 Nvidia H200 GPUs but there are parts like Intelligent Inference Scheduling which has been tested and run on single GPU.

Abylay Ospan who is one of Kernel maintainers:

The performance impact of GPU passthrough via vfio-pci in AI Linux (Sbnb Linux) is impressively low-averaging around 1-2% across a range of LLM models. This makes it a highly viable option for running accelerated inference inside virtual machines, enabling isolation and flexibility without compromising performance.