Setting Up Microsoft Azure for Parallel Calculations

Instructions for WEAP users who want to speed up large calculations by running scenarios in parallel on cloud computers ("workers") that WEAP starts and stops for you. No cloud or programming experience is assumed.  Total setup time: about 30-45 minutes, done once. See Parallel Calculations for details on all parallel calculation options.

What you are setting up, and what it costs

WEAP's parallel calculation feature splits a multi-scenario calculation across several virtual machines (VMs) in Microsoft's Azure cloud.  Each VM calculates some of your scenarios and sends results back; WEAP assembles them as if you had calculated locally.

You will create an Azure subscription (a billing account with Microsoft) and, inside it, a handful of free resources (a storage account for messages, a private network, and a credential that lets WEAP start VMs for you).  A setup script does almost all of it.

Typical costs, so there are no surprises:

WEAP deletes the VMs it starts when the calculation finishes.  The budget alert (Part D) is your safety net regardless.

Part A: Create your Azure subscription (one time, in the web browser)

You need a Microsoft account and an Azure subscription.  If your organization already uses Azure and can give you a subscription, use that and skip to Part B.  Otherwise:

1. Go to https://azure.microsoft.com/free and click Try Azure for free.

2. Sign in with any email address you control (a work address is fine).  If that address has no Microsoft account yet, the signup creates one -- just follow the prompts to verify the address.

3. Provide the requested identity verification and a credit card.  The card is for identity and for usage beyond the free credit; new accounts start with about $200 of 30-day credit, which is far more than this setup uses.

4. When the signup finishes you land in the Azure Portal (https://portal.azure.com).  Bookmark it.

That is the only part that cannot be scripted.  Everything else is automated below.

Part B: Run the WEAP setup script (one time, ~10 minutes)

The setup script is installed with WEAP (in the WEAP program folder, usually C:\Program Files (x86)\WEAP).  It creates everything WEAP needs and prints the values you will paste into WEAP.  It is safe to run more than once -- it skips anything that already exists.

1. In the WEAP program folder, double-click AzureEndUserSetup.bat.  (Advanced options live in the underlying AzureEndUserSetup.ps1 -- see the comments at the top of that file.  Note that the region should stay at the default eastus unless SEI has confirmed the WEAP worker image is available in another region: workers can only start in regions the image has been published to, and your quota and storage must be in the same region as the workers.)

2. If Microsoft's free Azure command-line tool ("az") is not on your PC, the script offers to install it first -- answer Y and accept the admin prompt.  (In the rare case it then asks you to reopen the window, do so and double-click the .bat again; it resumes where it left off.)

3. A browser window opens for you to sign in to Azure -- use the account from Part A.

4. If your subscription's processor-core quota is low, the script offers to request an increase for you -- answer Y.  (It is free, and Azure usually approves automatically within minutes.  If Azure insists on a human-reviewed request instead, the script prints the 2-click portal instructions -- see also Part E.  Note that free Azure accounts typically cannot increase their quotas until they are converted to Pay-As-You-Go accounts.)

5. Watch the green OK lines go by.  At the end the script prints a SETUP COMPLETE block with about ten labeled values.  Keep that window open (or re-run with -SaveToFile to also write them to a text file -- delete the file once you are done with Part C, because two of the values are passwords).

If the script warns that it could not create the service principal, you are in a company-managed Azure organization that restricts this step to the IT department.  The script prints exactly what to ask IT for; everything else will already have been created, and re-running the script after IT acts (or after they send you the two values) completes the picture.

Part C: Enter the values into WEAP (one time)

1. Start WEAP and open the area you want to calculate.

2. Choose the parallel calculation option (Area menu) and pick Azure as the Worker Location.

3. Click Edit next to the Azure account information to open the account dialog.  It has two sections, each with its own status line that turns green when the values entered have been checked against Azure and are valid.

4. In the Storage Container for Communications section, enter the Storage Account Name and Access Key from the script's summary, then pick the Container the script created (weapjobs).  Leave Path within Container blank.

5. In the Starting Workers from WEAP section, paste the Tenant ID, Application (Client) ID and Client Secret from the script's summary.  The Subscription ID then fills in by itself (if your account can see several subscriptions, pick the right one from the list), and the Resource Group is already set to weap-workers-rg -- leave it as is unless your IT department gave you a different resource group.

6. When both status lines are green ("validated"), click OK.  WEAP stores the values on your PC (the passwords encrypted); you will not be asked again.

You do not need to tell WEAP which worker image to start (the pre-built Windows disk with WEAP on it): the ID of SEI's shared image is built into WEAP, and WEAP can pick up a newer image ID automatically when SEI publishes one.  (Advanced users running a custom image can override this with the AzureImageId setting in WEAP.ini.)

Part D: Set a budget alert (one time, strongly recommended)

This emails you if spending passes limits you choose -- your safety net against a forgotten VM.

1. In the Azure Portal, search for Budgets (under Cost Management) and click Add.

2. Scope: your subscription.  Amount: e.g. $25 per month to start.

3. Add alert conditions at 50%, 80%, and 100% of the budget, with your email address.

4. Save.  (Azure's cost reports lag about a day; the alert emails do too.)

Part E: Request a bigger vCPU quota (only if the script's automatic request failed)

New subscriptions often start with a small limit on how much computing you may run at once (sometimes as low as 4).  Azure counts this limit in vCPUs (virtual processors).  WEAP's worker sizes use 1 vCPU per worker.  So to run 10 workers you want a limit of at least about 12-16.  The setup script normally requests this for you (Part B step 4); use this part only if that request failed or you skipped it.

Note: quota increases are only granted on pay-as-you-go subscriptions.  If yours is still the free trial, first upgrade it (Portal, Subscriptions, your subscription, Upgrade -- any remaining free credit is kept, and you still pay nothing beyond what you use), then request the increase.

Note: WEAP's discounted Spot workers (on by default) draw on a separate quota, Total Regional Spot vCPUs, requested the same way in the same screen -- but Spot increases are usually not approved automatically: when the request is declined, use the Create a support request button it offers. Quota support requests are free and usually answered within hours.  Until the Spot quota is raised, WEAP's quota message will tell you how many Spot workers fit -- or simply uncheck "Use Spot pricing" in the calculation dialog to run at the regular rate under the regular quota.

1. In the Azure Portal, search for Quotas and open it; choose Compute.

2. Find Total Regional vCPUs in the region the script used (default eastus), and also the per-family quota for WEAP's worker type -- Standard Fadsv7 Family vCPUs for the default worker size.  Both limits apply at once: workers only fit under the lower of the two, and WEAP's quota message names the one that is binding.

3. Click the pencil (or New Quota Request), enter the new limit (e.g. 16), submit -- for each of the two rows that is too low.

4. Requests for modest limits are usually approved automatically within minutes.  If a request is denied, a support ticket is offered in the same screen (in practice the regional requests tend to be approved automatically while the family requests more often need the ticket; it is free and usually answered within hours) -- SEI users can ask SEI support for help with the wording.

Part F: Your first parallel run, and good habits

1. Open your area, choose parallel calculation with Azure, pick the number of workers, and start.

2. When the run completes, WEAP deletes the workers it started.  You can verify in the Azure Portal: search "Virtual machines" -- the list should be empty a few minutes after the run.

3. Good habits: glance at Cost Management, Cost analysis after your first few runs so you know what a run costs for your model.  If a run crashes or you force-quit WEAP mid-run, the machines clean themselves up (each worker deletes its own VM when its work ends, WEAP removes leftovers the next time it runs, and every VM has a built-in 12-hour automatic shutdown) -- but a glance at the portal's Virtual machines list afterwards is a good habit, with the budget alert as the final safety net.

Troubleshooting

What got created, for the curious

Everything lives in one resource group named weap-workers-rg -- Azure's version of a folder.  Inside it: a storage account (the "mailbox" WEAP and the workers use to exchange jobs and results), a virtual network and security group (a private network with no inbound access from the internet), and a service principal (a robot login, limited to this one resource group, that WEAP uses to start and stop VMs).  Deleting the resource group deletes it all; nothing is installed on your PC.