Introduction
We have many different ways of delivering the Lateral API to clients who would like to install it in their own environment. One of those is as an Azure VHD for deployment to Azure VMs. In this post I will cover how to create a VHD that is fully compatible with Azure from an Ubuntu Cloud Image base.
I found the Azure documentation to be a bit outdated. In fact in various places it didn't work at all. So I thought I'd try to create a version that at least works for Mac OS X users. Well, that works until it itself becomes outdated and stops working. At which point I or hopefully someone else will take it upon themselves to write an updated version.
There were also quite a few gotchas that I will explain in due course. One that deserves an honorable mention is the fact that the size of the Azure VHD must be aligned to the nearest whole MB otherwise the image will be rejected after upload. I mean, seriously Microsoft??
Dependencies
There are several dependencies required before we can start:
* VirtualBox with `VBoxManage` and `VBoxHeadless` on your path
* Go for the `azure-vhd-utils-for-go`
* Azure CLI for interacting with Azure from the command line
* `qemu` for `qemu-img` used in `create-vhd` script to resize and convert images
* `coreutils` for `gtruncate` used by the `create-vm` script
* `cdrtools` for `mkisofs` used by the `create-vm` script
* `gawk` used by the `create-vhd` script
You can use Homebrew to install the last 4 with the following command:
<pre>
brew install qemu coreutils cdrtools gawk
</pre>
Next you need two scripts; `create-vm` and `create-vhd`. The `create-vm` script was modified to work with OS X from the example written by Scott Moser. Thanks Scott! Have a read through the comments of the two scripts to get an idea of what they do. Then download them to a location of your choosing and ensure they are executable:
Creating the VM
Use the `create-vm` script to create a VM in VirtualBox from the Ubuntu 14.04 image:
<pre>
./create-vm my-trusty-vm
</pre>
If you want to use a version of Ubuntu other than 14.04, just modify lines 8-9 of `create-vm` to point to the version you want.
Once that has completed, you can start the VM in headless mode:
<pre>
VBoxHeadless --startvm my-trusty-vm
</pre>
Now you can SSH into the instance from port `2222`:
<pre>
ssh -p 2222 -A ubuntu@localhost
</pre>
The password is set at line 35 of `create-vm` to `ubuntu`. We strongly recommend changing this to something secure or even better setting up key based authentication.
When you're in the instance there's a bit of setup required for it to work with Azure. See here for some more details about what's being done here:
<pre>
sudo apt-get update
sudo apt-get install -y linux-image-virtual-lts-xenial linux-lts-vivid-tools-common hv-kvp-daemon-init
sudo reboot
</pre>
After the system reboots, edit `/etc/default/grub` in a text editor, find the variable called `GRUB_CMDLINE_LINUX_DEFAULT` and change to:
<pre>
GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0,115200n8 earlyprintk=ttyS0,115200 rootdelay=300"
</pre>
Then update grub:
<pre>
sudo update-grub
</pre>
And install the azure agent:
<pre>
sudo apt-get update
sudo apt-get install walinuxagent
</pre>
Now that's all installed you can do whatever you need to get the VM up and running how you like it. Install packages, clone your code to the VM and set it up to start at boot, etc.
Preparing the VM for deployment
This step is important to get the system running successfully on Azure. When the instance is in a good state and you don't want to make any more changes, run the following:
<pre>
sudo waagent -force -deprovision
history -cw
logout
</pre>
And safely shutdown the VM:
<pre>
VBoxManage controlvm my-trusty-vm poweroff soft
</pre>
Now you can use the `create-vhd` script to create an Azure VHD from the VDI. Note this will require about 60 GB free disk space:
<pre>
./create-vhd my-trusty-vm
</pre>
This will take a little while to complete. The script takes care of the resizing of the VHD to the nearest MB as required by Azure so you don't have to worry about that!
Uploading the VHD
Once completed we are ready to upload the VHD to Azure. The documentation from Microsoft says to use the `azure-cli` to upload the image. However, trying to use it caused this error for me. This response from March 2016 basically says it's being looked in to and in the meantime we should use the `azure-vhd-utils`. Which does work! So to install that run:
<pre>
go get github.com/Microsoft/azure-vhd-utils-for-go
</pre>
And then you can run the tool to upload the image:
<pre>
azure-vhd-utils-for-go upload --localvhdpath my-trusty-vm.vhd --stgaccountname NAME --stgaccountkey KEY --blobname my-trusty-vm
</pre>
Replacing `NAME` with the storage account name and `KEY` with the storage account key. You can get these from the Azure Portal. `azure-vhd-utils-for-go` is good because it doesn't upload empty space. So if the VHD is 30 GB on your disk it will only upload the parts of the image that have data in. This speeds up the upload process greatly.
Now you have to create an image from the Azure VHD stored in blob storage:
<pre>
azure vm image create --blob-url VHD_URL --os Linux my-trusty-vm ''
</pre>
Replacing `VHD_URL` with the URL to the Azure VHD in the storage account. You can get the full URL to the VHD by navigating to it using the Microsoft Azure Storage Explorer, right clicking the file and choosing "Copy URL to Clipboard". The `''` is required because the CLI is expecting a file to be uploaded there. If you leave it as an empty string it works and takes the VHD from the storage account.
Lastly, you can create a VM from the image:
<pre>
azure vm create --userName ubuntu --password 'Pas$word1' --vm-size "Standard_A4_v2" --ssh --location "West Europe" "MyTrustyVm" my-trusty-vm
</pre>
Once that is completed, you're done! You can SSH into the instance that was created from the Azure VHD now. Note that the password specified in the above command is required but doesn't actually do anything. The password to the VM was set earlier by the `create-vm` script to `ubuntu`. Important: Enable key based authentication now otherwise you have a very insecure instance running!
Conclusion
I hope that this post helps people in the future who like me found the Azure documentation to be a bit broken and confusing.
References
* Prepare an Ubuntu virtual machine for Azure
* Example of using Ubuntu cloud images with virtualbox
* Problems preparing a disk image for upload to Azure
