The 6 Most Important Things I learned in my 6 Months using Serverless AWS CLOUD LAMBDA Finding the right tools is of utmost importance in the world of serverless. October served as an eye opener to me and my company. After the Serverlessconf tour, I decided there and then that my company would operate on Serverless. The first two months were a nightmare as we struggled to familiarize with the trend. Six months down the …
Amazon Dynamo was created in 2004 to scale the growth of Amazon’s Oracle database infrastructure. The aim behind its creation was to meet the business’s requirements (scalability, performance, and reliability).
In 2012, the availability of DynamoDB as a fully managed NoSQL data service was announced by AWS. AWS promised that it would have seamless scalability.
Why choose DynamoDB?
I interviewed a number of developers and engineers about their experience using DynamoDB. Even though this database service has many success stories, it has left behind many failed implementations. To fully understand why DynamoDB succeeds in some areas and fails in others, you first have to learn about the tension between two of its greatest promises- scalability, and simplicity.
DynamoDB is simple to use until it refuses to scale
Throwing data in DynamoDB is the easiest thing you can ever do. It is less complex as you don’t have to be worried about logging in and setting up a cluster- all thanks to AWS. To start operating this service, you just turn a knob, look for an SDK and sling JSON.
However, as much as DynamoDB is simple to interact with, designing its architecture is a difficult task. It works well during retrieval of individual records that may be based on key lookups. Where complex scans and queries are involved, there is a need to carry out indexing carefully. This is a must even if the amount of data isn’t huge and you are familiar with the design principles in NoSQL.
Most developers know a lot about classic relational database design but not much about NoSQL. A combination of inexperienced developers, the absence of a clear plan on modeling a dataset in DynamoDB and a managed database service is a recipe for failure.
First Law of DynamoDB
The first law of using DynamoDB is to assume that its implementation will be harder compared to employing a relational database you are well-versed with. At a small scale, a relational database will accomplish each of your need. Setting it up will initially take a long time compared to DynamoDB. However, the well-established SQL conventions will save you a lot of time in the long run. This is far from the assumption that DynamoDB technology is awful. It is because it is new to you.
DynamoDB can be scaled – until it’s not simple
For this article, I interviewed a few DynamoDB happy customers. DynamoDB promises great performance at an infinite scale which is only limited by the size of the AWS cloud. The customers are right in the center of doing key-value lookups on well-distributed records, avoiding complicated queries and limiting hotkeys.
DynamoDB is well-known for dealing with hotkeys and this is explained in detail in the DynamoDB developer’s guide documentation. Although it can scale indefinitely, data is not stored on a single server. As it grows larger, it is divided into chunks, each on a different partition.
Despite DynamoDB being able to scale indefinitely, your data is not stored on one, ever-expanding server. What happens is that the capacity of a single DynamoDB shard is divided into parts as your data increases. Therefore, each part lives on a different shard.
If you have a hot key in your dataset, you must ensure that the allocated capacity on your table is set high enough to handle all the queries.
With DynamoDB, you can only provision its capacity at the entire table level. You cannot provision its capacity per partition. By use of a fairly wonky formula the capacity is divided up among partitions. Consequently, your capacity for reading and writing on any record becomes smaller. If your application has too many RCUs on one key, you can do three things; over-provision all other partitions which are rather expensive, generate errors or decrease access to the key.
One thing to note however is that DynamoDB is not suited to datasets that are a mixture of hot and cold records. But at a large scale, each dataset has a similar mixture. You can split the data into tables, but you will end up losing the scalability advantage of DynamoDB.
A recently published article on “The Million Dollar Engineering Problem” showed how Segment decreased their AWS bill. It did it by fixing the DynamoDB over-provisioning. Alongside the article was a heat map graphics that showed the partitions that were troublesome.
The graphics originate from AW’s internal tools. Their strategy for blocking the troublesome keys was to wrap DynamoDB calls in a try. ‘Segment then had to battle the hotkeys problem, and this is where the simplicity and scalability factors came in.
Designed as a black box, DynamoDB has very few user-accessible controls. When starting, it is this aspect that makes it easy to use. But at the production scale, you need more insight into your data misbehavior.
Second Law of Dynamo
The second law of DynamoDB states that DynamoDB’s usability, at a massive scale, is limited by its own simplicity. The problem here is with what AWS has chosen to expose, not Dynamo’s architecture. Its failure to backup 100TB DynamoDB data was the leading reason why Timehop moved off the service altogether.
What if you can’t use DynamoDB?
First, Let us look at the advantages and disadvantages of using DynamoDB
- Handles huge amounts of workloads.
- Enables you to redesign many applications.
- Enables you to store state in a K/V table.
- Enables use of event-driven architecture to suit your desires.
- Only suitable for small scale.
- You can’t redesign all the applications.
Just because you can use DynamoDB does not mean you should use it. Using Dynamo without fully understanding will make you end up spinning your wheels through several code rewrites before you land on a solution that works .
Third Law of DynamoDB
The third law of DynamoDB states that business value trumps architectural design each time.
This is the reason why the various developers I interviewed abandoned NoSQL to provide solutions for both small and middle-sized businesses. It is also the same reason why Timehop moved from DynamoDB to Aurora. This also explains why DynamoDB has lots of case studies from happy customers globally.
The Introduction of WhynamoDB
Amazon will at some point announce the release of WhynamoDB. The decision tree below will guide you on this new service.
Reasons Why DynamoDB is Not for Everyone AWS CLOUD LAMBDA Amazon Dynamo was created in 2004 to scale the growth of Amazon’s Oracle database infrastructure. The aim behind its creation was to meet the business’s requirements (scalability, performance, and reliability). In 2012, the availability of DynamoDB as a fully managed NoSQL data service was announced by AWS. AWS promised that it would have seamless scalability. Why choose DynamoDB? I interviewed a number of developers and …
Practical Tips and Tricks on How to Use Typography in UI Design AWS CLOUD LAMBDA Introduction Typography is the most challenging part of the UI Design despite having it in various forms from time immemorial. Owing to its long existence, we have theories, rules, and practices which we must keep up with. This article presents some practical typography tips and tricks to use in your projects. Practical examples instead of theories I will not dwell …
How AI, VR, and Big Data Will Transform the Real Estate Industry by 2020 AWS CLOUD LAMBDA I came to learn of AI through my colleagues. I heard them speaking of a new feature that was about to be rolled out by our company. Throughout the conversation, I was blown away as it looked like the perfect thing. There was a lot of details on AI, and it had been on the news already. It …
Cheat Sheets for AI, Machine Learning, Neural Networks, Big Data & Deep Learning AWS CLOUD LAMBDA I have been collecting AI cheat sheets for the last few months, and I’ve been sharing them with friends and colleagues from time to time. Recently, a lot of inquiries concerning the same sheets have been made, and so I’ve decided to organize and share the entire collection of the sheets. In this article, I have added descriptions and …
16 Things to Avoid When Writing for Your UI AWS CLOUD LAMBDA You write for your UI with an intention of making it useful to people, right? Therefore, you should ensure that it is simple, clear, and concise to enhance usability. Below are 16 things to avoid when writing. Technical Terms and Specific Words Use of words that are meant for specific professions should be avoided. Words that are specifically meant to describe certain things …