Detecting incast congestion of RDMA flows

The recently released open source GitHub project, ai-monitor, is a proof of concept demonstrating that real-time sFlow telemetry can detect and characterize incast RoCEv2 RDMA flow collisions between GPUs during AI training. 
docker run -p 6343:6343/udp -p 8008:8008 sflow/ai-monitor
The pre-built sflow/ai-monitor image is the quickest way to try out the software. The diagram at the top of this page shows test RDMA flows crossing a 4 leaf / 2 spine test network to generate an incast collision. The flows are displayed using the trace-flow application that is included in the image.

The ai-monitor application uses packet samples to detect when GPU port is oversubscribed:

  1. Detect when an leaf switch egress port utilization exceeds 80%
  2. Count the number of sources sending traffic to the port - single flows are ignored
  3. Trace each of the flows back across the topology to its source GPU

The instant the threshold is crossed, the following incast_congestion event is generated by querying information maintained within the sFlow-RT analytics engine. Real-time elephant flow monitoring in AI / ML fabrics describes how standard sFlow from data center switches streams real-time telemetry that can be used to detect elephant flows with a few milliseconds.

{
 "event":  Continue reading

What a ‘Kill Switch’ Actually Means for Your Agent Fleet

In short, in the five weeks to 27th October 2026, Okta, Satya Nadella, the Cloud Security Alliance and Salesforce each put the words “kill switch” or “emergency brake” next to AI agents, and the AI Kill Switch Act, introduced in the US House on 23rd July, would require the largest model providers to be able to throttle, suspend or shut their systems down on order. Every one of those switches stops something, and none of them stops the same thing. Pull the switch on an agent in your cluster and read what is still true a minute later: the Deployment has replaced the pod, the callers are retrying, the open sessions are still open, and the key in the agent’s environment still works. A kill switch for a fleet is a set of cuts at points the agent does not control, and the only way to know which ones you hold is to pull them. This post reads the bill and the UK NCSC’s advice for what a stop has to do, runs a drill through the five switches teams reach for, then says what a governance layer like Tigera Lynx can and cannot do about it. It is written Continue reading

Space Engineers: Panduan Merancang Kendaraan Antariksa dan Stasiun

Dalam industri permainan simulasi dan rekayasa modern, Space Engineers menjadi salah satu game yang paling diminati hingga saat ini. Dengan fokus pada kreativitas, teknik, dan eksplorasi luar angkasa, Space Engineers memungkinkan para pemain untuk tidak hanya merancang kendaraan antariksa yang kompleks tetapi juga membangun stasiun ruang angkasa secara mandiri. Artikel ini akan membahas secara komprehensif cara merancang kendaraan antariksa dan membangun stasiun sendiri di Space Engineers dengan panduan yang mudah diikuti dan relevan dengan perkembangan terbaru dalam game tersebut.

Pendahuluan: Mengapa Space Engineers Penting untuk Para Pecinta Antariksa?

Space Engineers adalah game sandbox yang memadukan simulasi rekayasa dan eksplorasi luar angkasa dengan tingkat kebebasan yang sangat tinggi. Hingga saat ini, game ini telah berkembang dengan banyak tambahan fitur dan optimasi, mendukung pengalaman pengguna yang semakin realistis dan menantang. Para pemain dapat menciptakan berbagai jenis kendaraan antariksa mulai dari roket, pesawat luar angkasa, hingga kendaraan darat yang bisa menjelajahi planet. Selain itu, konsep membangun stasiun luar angkasa yang kompleks menjadi daya tarik utama yang memungkinkan pemain mengasah keterampilan teknik dan kreativitas mereka.

Merancang Kendaraan Antariksa di Space Engineers

1. Memahami Dasar-dasar Desain Kendaraan Antariksa

Sebelum mulai merancang kendaraan antariksa, penting memahami elemen dasar yang harus diperhatikan, seperti:

Astroneer: Panduan Menjelajahi Planet dan Membuat Koloni Efisien

Pendahuluan

Astroneer tetap menjadi salah satu permainan eksplorasi dan simulasi luar angkasa favorit bagi para penggemar di periode terbaru. Dengan mekanisme gameplay yang menuntut kreativitas dan manajemen sumber daya yang efisien, Astroneer memberikan pengalaman unik dalam menjelajahi berbagai planet asing dan membangun koloni dari nol. Artikel ini akan membahas secara lengkap cara menjelajahi planet dan membangun koloni dengan sumber daya terbatas dalam Astroneer, yang tentu sangat relevan bagi para pemain saat ini yang ingin memaksimalkan pengalaman mereka di game ini.

Memahami Konsep Eksplorasi dan Sumber Daya di Astroneer

Salah satu kunci dalam Astroneer adalah pengelolaan sumber daya yang efisien. Saat ini, para pemain dihadapkan pada tantangan untuk menjelajahi berbagai planet dengan kondisi yang berbeda-beda, menggunakan alat dan kendaraan yang terbatas. Pengetahuan tentang sumber daya utama seperti Resin, Compound, Laterite, dan Organic sangat penting karena bahan-bahan inilah yang akan digunakan untuk membuat berbagai alat dan infrastruktur dasar.

Resin biasanya digunakan sebagai bahan bangunan dasar untuk membuat lander, platform, dan berbagai komponen mesin. Sementara Compound diperlukan untuk membuat paket-paket energi dan alat dasar. Oleh karena itu, prioritas pertama saat mulai bermain adalah mencari dua sumber ini dan mulai membangun basis yang kokoh.

Strategi Menjelajahi Planet dengan Sumber Daya Terbatas

Menjelajahi planet dalam Continue reading

Getting Structured Output from Network Devices with Ansible

Getting Structured Output from Network Devices with Ansible

This post is part of an ongoing series on Ansible, and you can find all the other posts in the series here.

In the previous parts of this series, we built an OSPF network with Ansible, and then wrote a few checks to make sure it's healthy. One of those checks counted the OSPF neighbors on each router, by running show ip ospf neighbor and counting the lines that contain the word FULL. It works, but it's a bit broken. We're treating the output as plain text and searching through it, so we're relying on the router always formatting the output in exactly the same way.

The output of a show command is designed for humans to read, not for machines. What we really want is structured data, such as a list of neighbors, where each neighbor has its own neighbor ID, state, and interface that we can work with directly, just like the facts and variables we've been using throughout the series.

In this part, we'll look at how to parse command output into structured data using the cli_parse module and a library of ready-made templates called ntc-templates. We'll parse show ip ospf neighbor, compare the plain Continue reading

Pre and Post Checks with Ansible

Pre and Post Checks with Ansible

This post is part of an ongoing series - see all the posts below.

Ansible Course

In the previous part, we wrote two checks for our OSPF network. One makes sure every router has all of its OSPF neighbours, and the other makes sure every router can reach every other router's loopback. They're useful on their own, but they're even more useful when you run them around a change.

The idea is that before making a change, you check that the network is healthy. If it isn't, you stop, since there's no point making a change on a network that's already broken. It also means that if something goes wrong later, you know it wasn't already broken before you started. Then, after the change, you run the same checks again to make sure everything is still healthy.

Most of us already do this manually. Before a change, we log in and save the output of a few show commands, and afterwards, we run them again and compare. In this part, we'll let Ansible do it for us, by reusing the playbooks we've already written, rather than copying them into every change playbook.

We're continuing with the same lab and Continue reading

4 real? Tracking exploitation and patching of eMudhra’s weak keys

On October 19, 2026, we learned about eMudhra’s eight shockingly weak keys: where eMudhra was trusted to pick a 115-digit number for their virtual combination lock protecting the Internet, they picked the number 4. Not even four digits, just the number 4. The other seven keys were similarly weak: 56, 66, 115, 155, 156, 230 and 237.

The next day, we wrote a blog post on the immediate action we took to mitigate this leak, and what customers needed to do on their end to stay secure. Now, we are sharing a more in-depth follow up on the evidence we’ve seen of exploitation, and the progress seen on visitors patching their systems.

Let's start with the scale of the problem. A certificate authority (CA) with weak keys this serious is unprecedented, and the potential consequences are significant. While there have been plenty of compromised certification authorities in the past (such as DigiNotar, TÜRKTRUST, and TrustWave), access to their keys has always been contained to a degree; these incidents gave a limited set of attackers or organizations the ability to issue fraudulent certificates. The eMudhra incident is fundamentally different; the affected intermediate private keys could be recovered almost Continue reading

Cloudflare services for private origins is now generally available

Today, Cloudflare services for private origins is generally available. Point a proxied DNS record at a private IP, and Cloudflare reaches the origin over your existing Cloudflare One connection. Your origin stays private, while web application firewall (WAF), caching, bot management, and Workers work just as they do for public origins.

In June, we launched this capability in closed beta to address a problem we kept hearing from customers: Their internal APIs, AI agent backends, MCP servers, and operational tools weren’t built to face the Internet, but those services still need to accept requests from partners, customers, or agents over the Internet. Before the beta, putting Cloudflare in front of these services meant exposing a public IP, opening a firewall port, or running cloudflared alongside the origin.

Those requirements often left private applications without the same protections and controls they used for public applications, including WAF, bot management, rate limiting, caching, traffic acceleration, rewrites, and Workers.

What is generally available today

During the closed beta, eligible customers requested access through their account team. Today, we’re opening access beyond that beta group.

Point a proxied A or AAAA record at a private IP, and Cloudflare’s application services work Continue reading

Worth Reading: Network Labs on a Budget

The Packet Pushers newsletter dropped into my Inbox this morning, and it included a lovely Network Labs on a Budget article by Suresh Vinasiththamby, explaining how you can build a virtual lab on a mini-PC (exactly what I’m using for my netlab server) using EVE-NG, CML, or the containerlab/netlab combo.

Keep reading

On a tangential note: I’m busy getting various netlab features to work with netmiko, so something had to give (= blog posts).

When Novel Becomes Normal

Imagine your product strategy team gets stuck on a positioning problem. Somebody opens an AI chat window and asks for an unconventional angle. The model comes back with a concept nobody in the room has thought about in years. Maybe it’s the Lindy effect or Chesterton’s fence or something borrowed from evolutionary biology. The room loves it. It feels fresh and a little clever. Three months later a competitor walks onto a keynote stage with the same framing. How did that happen? Did someone leak the deck?

You’ve probably felt a smaller version of this. You learn a new word and it shows up in three articles that same week. The linguist Arnold Zwicky named this on Language Log in 2005. He called it the frequency illusion, though you may know it by its folk name, the Baader-Meinhof phenomenon. Once you notice something, you think it’s everywhere. Most of the time it isn’t. Your attention changed. The world didn’t. When the coincidences really start piling up, some people reach for something grander. Great minds think alike. Ideas are in the air. Maybe there’s a universal consciousness out there that we’re all dipping into. Or maybe it’s as bland as Continue reading

Cloudflare DDoS Threat Report 2026 Q3: More swarms, fewer giants

Welcome to the 26th edition of Cloudflare's DDoS Threat Report. This edition covers the third quarter of 2026 — July, August, and September — and returns the series to its quarterly cadence following the combined H1 2026 volume. The analysis is produced by Cloudforce One, Cloudflare's Threat Intelligence organization, providing a comprehensive analysis of the evolving threat landscape of Distributed Denial of Service (DDoS) attacks based on data from the Cloudflare network.

Key insights

  1. Short-burst, low-bandwidth attacks lasting under 10 minutes and 1 Gbps surged by 208%. Conversely, hyper-volumetric attacks exceeding 1 Tbps plummeted by nearly 69% from Q2, possibly a reflection of law enforcement actions against four major Internet of Things (IoT) botnets earlier in the year. However, variants continue to emerge, proving that IoT botnets remain a durable, high-volume threat to network infrastructure.
  2. Threat actors dramatically shifted their reflection and amplification tactics, abruptly abandoning CLDAP floods, which collapsed by 99.5% quarter-over-quarter. Instead, attackers pivoted heavily toward other connectionless protocols, driving a 215% surge in DNS amplification and a 122% increase in general UDP floods.
  3. Attack targets broadened significantly across both industries and geographies, with the United States reclaiming the most-attacked position from China. Canada and Israel Continue reading

Fabric Topologies for GPU Clusters

In part five of a series on AI data center networking, various interconnected networks are discussed, highlighting the differences between traditional and AI clusters. Key topics include back-end and front-end networks, the folded-Clos topology, adaptive routing, and the significance of switch design in addressing network congestion. Dragonfly topology is also mentioned for hyperscale systems.

The post Fabric Topologies for GPU Clusters appeared first on /overlaid.

Booth Demo

Quickly engaging an audience is a challenge at a trade show - all Ethernet switches look pretty much the same and they don't do much other than blink LEDs. Real-time sFlow telemetry is a core feature of switches that can be used to create intriguing animated dashboards that draw customers into the booth by demonstrating how the switch addresses their needs.

The example above uses the open source sFlow-RT booth-demo application to demonstrate a 400G data center switch carrying RoCEv2 test traffic. The demo shows a series of RDMA write operations in the Traffic Trend chart. The Switch Port Traffic Flows diagram shows the paths traffic takes as it flows through the switch. In this example, a NCCL ring topology results traffic flowing from port to port as it goes around the logical ring of GPUs.

The quickest way to set up the demo is to run the a pre-built sflow/booth-demo image on a laptop connected to the booth display screen. The Traffic Trend chart is easily customizable to show different flow attributes that are relevant to the demo and the title can be changed to include branding and messaging. If more extensive customization is required, clone the GitHub project Continue reading

1 2 3 … 3,918