AWS has spare compute resources, which they make available via auction. The prevailing price for these resources (spot price) fluctuates with market conditions, but is usually significantly lower than the fixed price of On-Demand (guaranteed) compute resources.
In exchange for deeply discounted prices, Spot Instances can disappear before you are done using them if the spot price increases. In practice, this happens very rarely.
Also, spot instances take longer to start, as they must first go through a bidding process on the spot market. More details about spot instances and the spot market are here.
When you start a Mortar job on a new cluster, Mortar by default requests an On-Demand Elastic MapReduce (EMR) cluster for you. On-Demand EMR clusters have fixed costs per hour, launch quickly, and can be counted on to stay running until your job is complete.
Alternately, you can request for Mortar to use a Spot Instance cluster instead. Spot Instance clusters also use EMR, but they are powered by Spot Instances instead of On-Demand instances. Their hourly price fluctuates with supply and demand on the spot market, but they are frequently around 70% cheaper than On-Demand clusters.
There are two tradeoffs to be aware of:
For Web Projects, when you run your job just check the Use Spot Instances checkbox:
If you hover over the Use Spot Instances checkbox, you can also see the current price for Spot Instance Clusters.
For Mortar Projects, add the
--spot switch to your command:
$ mortar jobs:run pigscripts/my-sample-project.pig --clustersize 3 --spot
We suggest using them everywhere except time-sensitive production jobs. As long as the (very rare) case of a cluster stopping before your job is finished is acceptable, you will get great value from Spot Instance clusters.
We frequently see them cost roughly $0.14 / node-hour, though the price can change as the spot market fluctuates.
Mortar passes through the exact AWS cost with no upcharge. As with On-Demand clusters, this cost has an EMR component and an EC2 component.
The EMR component stays fixed at $0.088 / node-hour for m1.xlarge instances in US-EAST-1.
The EC2 component fluctuates based on spot price. The current spot price for m1.xlarge in US-EAST-1 can always be found on the AWS site or by hovering over the Use Spot Instances checkbox when running your job in Mortar.
As an example, at time of press you would pay roughly $0.14 per node-hour: $0.088 for EMR and $0.05 for EC2. This is almost 70% cheaper than On-Demand clusters.
When Mortar bids on the spot market, we use exactly the On-Demand price ($0.35 / node-hour). This is the maximum spot price you would pay per node-hour.
Bidding this price ensures that instances are provisioned quickly. In practice, we almost always see an accepted price of $0.06 / node-hour or less.
While interruption is always a possibility, at the price we bid it is very rare. That said, if your work is time-sensitive, you may want to continue to use On-Demand clusters to ensure it finishes on-time, every time.
Mortar continuously monitors your Spot Instance clusters, searching for interruptions. If your spot instances are reclaimed by AWS, Mortar will stop your job automatically.
Per standard AWS pricing, if your cluster is interrupted, you will not be billed EC2 instance costs for the last hour of usage, though EMR costs will still apply.
On rare occasions, the spot price rises above the on-demand price. If you try to launch a cluster during this time, Mortar will wait for 30 minutes to see if the price drops. If it does not, Mortar will switch your Spot Instance cluster to an On-Demand cluster, ensuring you can run your job.