Azure Virtual Machine Scale Sets let us create and manage a group of VMs, then change the instance count as demand changes. These are my notes on the pieces around that: keeping instances available, resizing them, and getting a replacement machine configured without logging into it.

The design fork

The main question is whether you need virtual machines at all:

  • Choose PaaS/serverless if the platform can run the workload and you mostly care about the application.
  • A managed container service may fit if the workload can be packaged in an image. Containers on their own don’t remove the job of managing the hosts.
  • Choose VM Scale Sets when you really need VM-level control, custom agents, legacy software, or special networking/host assumptions.

Scaling a VM up and scaling a service out are different moves. Resizing a VM can help a single overloaded machine, but it usually creates downtime and does not improve fault tolerance. Adding instances behind a load balancer improves capacity and resilience, but only if the application is stateless or handles shared state properly.

Availability Sets

  • Two or more VMs in an availability set meet the placement requirement for the 99.95% VM availability SLA. That is a platform commitment, not a promise that our application will be available for that percentage of time.
  • An availability set distributes VMs across fault domains (shared power/network hardware) and update domains (groups that can restart together during maintenance). It doesn’t mean every VM gets its own rack.
  • This arrangement would cover you for maintenance & local failures.
  • This arrangement on its own won’t help with regional/data center failures (since your set of machines are in the same data center).

Microsoft’s availability set overview covers the fault/update domain behavior and recommends comparing this with flexible scale sets for new deployments.

Availability Zones

  • Availability zones are separate physical locations within a region, each with independent power, cooling and networking. Deploy instances across zones to survive a zone failure; assigning a single VM to a zone doesn’t duplicate it.
  • The data centers are connected by high speed redundant links.
  • They are not available in every region.

Azure Traffic manager

  • We could use Traffic Manager to direct clients to application endpoints in two regions. It doesn’t create network connectivity between those regions.
  • Azure Traffic Manager operates at the DNS layer
  • Directs incoming DNS requests based on the routing method of choice. (Choices are: Priority, performance, geographic, weighted round-robin, subnet, and multi-value and you can combine them)
  • With performance routing, the answer is based on measured network latency, which isn’t necessarily the geographically closest endpoint. Cached DNS answers also mean a change won’t move every client immediately.

Managed Disks Storage

  • Slight side track, but when designing IaaS remember this is the way to go, don’t use the classic storage account for this.

Resize VMs for scale?

  • You can resize Azure VMs on the fly, either in the console or via PowerShell.
  • They will go down while you do though, in the lab this was only about a minute but its hard to say if machines doing real things might take longer.

Scale Sets

  • Keep in mind, IaaS isn’t the best for horizontal scale. PaaS is designed to solve the scale issues for you and is the way to go if possible/practical for the use case.
  • But, “Scale sets” are what you want if you need to handle scale.
  • You’ll want a “Virtual Machine Scale Set template”
  • Configure the scale options, configure a load balancer and off you go.
  • The Azure marketing pitches it like “Virtual Machine Scale Sets are elastic and designed to support your scale-out workloads, including stateless web front ends, container orchestration, and microservices clusters. Azure Kubernetes Service and Azure Service Fabric run on Virtual Machine Scale Sets”

Before relying on autoscale, try creating one instance from scratch. Can it fetch its configuration, reach its dependencies and pass the health check without someone logging in? Scaling out an incomplete image just gives us more broken machines.

Deployment Automation

  • It is all about ARM (Azure Resource Manager) templates.
  • I’ll do a separate post to summarize ARM templates because there is too much content but important to know a few things here…
  • When you deploy a template, Resource Manager converts the template into the individual REST API operations needed to achieve the thing you’ve declared in your template.
  • Visual Studio makes this easy. It has “validate” and “deploy” options built right in and produces a skeleton template for the most useful scenarios. Today, I would also consider Bicep or Terraform for readability and reviewability. Start with the current Azure Resource Manager template documentation, Bicep documentation, or Azure quickstart templates.
  • Small tip: Be careful of PowerShell windows that have popped up behind visual studio when testing deployments. (Your deployment might be waiting for input from you).
  • Also, for what it is worth, to test the declarative nature of ARM templates I deleted the NIC from a machine configured via an ARM template, then reran the template. The NIC was put back in place, and it was still the same machine (it didn’t burn the old and make a new one) because I left a ping running inside it to confirm. I did find the machine couldn’t reach the Internet anymore though, even after getting its NIC back. I used “redeploy” to move it to another host and it had Internet access again.

Quickstart Templates

Policy Restrictions

  • If you’re worried that all this scale, redundancy and automation might land you in hot water with machines deployed in a prohibited region consider an Azure policy that describes the regions your are allowed in, attach it to the subscription and you’ll be protected from accidents of this nature.

Custom VHD Templates

  • Last thing, sort of related to scale. You can work on your own VHD images. Just build them in Hyper-V locally, generalize them and use AzCopy/Storage Manager to get them into your account. You can store them on blob storage in a storage account.

References