@TobiasTernent

GitHub: Tobias-Ternent
Showing posts with label aws. Show all posts
Showing posts with label aws. Show all posts

Thursday, 23 June 2016

AWS storage

AWS offers 3 main ways for data storage:

S3 
Simple Storage Service (S3) - data is held in the cloud, in a "bucket". Inside a bucket, files and sub-directories can be created. The point of this type of storage is to write once, read may times perhaps from different services (EC2 instance, the Internet, etc). They can also be deleted easily. Data are automatically backed up and have a 99.9% uptime.

EBS 
Elastic Block Storage (EBS) - these are volumes that can be added onto EC2 servers, think of them like hard drives. They can be scaled up/down, added, removed, backed-up as needed per instance. 

Glacier 
The Glacier is for remote "off-site" backups. Write once, read (almost) never. This should be for long-term archival of data, e.g. that was present in S3 for a long time, and will never normally be used again, save for restoration of backup data. Glacier prices are cheaper than S3 for storage, but have retrieval costs.

DevOps and AWS

DevOps is the practice about Development and Operations. It consists of a typical 'toolchain' of many tools that together describe how you should work on a software project. The main parts are as follows:
  • Code – Writing the source code with a programming language. E.g. Java, Python, etc. This performed using an IDE, e.g. Idea IntelliJ for Java.
  • Manage – Work has to be managed somehow, i.e. short, medium, long-term aims, sprints, issues, tasks, stories, bugs, milestones, etc. This can be managed with tools, e.g. JIRA.
  • Store – The source code needs to be stored somewhere, e.g. using a Git repo hosted on GitHub, BitBucket. Similarly, built libraries will need to be stored and available somewhere, e.g. Nexus.
  • Build/Package & Test/Release – The software will need to be built. Changes made to a library/repository that is used by another might have consequences. Or simply work done is ready for QA testing before production deployment. There are tools available to help automate and manage all of this for building software automatically according to your needs, e.g. TravisCI, Jenkins, TeamCity.
  • Deploy – Software that has been built will then need to be deployed somewhere, i.e. for a test , development, or production environment. These environments and builds should be monitored too. Tools can be setup to help manage this as well.

AWS has solutions for all of these aspects, much like it does so for the infrastructure side of things. Examples of these are:

There is no obligation to use any of AWS' services, and you can use as many or as few of them as needed. They are paid-for services, but they all tend to have free limited usage still either during the first 12 months or over time, so you'll only pay for how much you go over the free provision, or plan to use in terms of numbers of commits, builds, etc. If you're considering to use AWS for these features, each service should be considered so that a good, timely decision is made so progress is not impeded and productivity is maintained.

DevOps is similar to being 'agile', i.e. employing methods and tools to increasing quality and quantity of work. Agile is more of a way of thinking, DevOps actually requires organizational changes - actual methods in the working culture and for delivering software.
DevOps can be implemented from the ground up to avoid future upheaval. It's really about applying agile principles, and there is no reason not to do this.

I'm sure we would all agree that it would be best to follow industry 'best practices' for tools and solutions. If other people have already performed analysis, and have detailed what options we have, and what the best options are for each use-case, then there is no reason to reinvent the wheel or procrastinate on matters. What matters most is finding a good solution quickly, and going with it. Even though this might not be the most perfect solution, perfect is the enemy of good.

An example of this would be for choosing a cloud platform: Google Cloud vs AWS. Both have almost identical offerings. Does it really matter which provider to go with? In general terms, no, as long as you go with something that fits what you want to do now, with a capacity for the future in mind. Such a decision does need to be agonised over in order to make rapid progress.

Website solution stacks

In summary, the normal options about how to create a website on a platform are:
  1. Traditional LAMP stack, or a variant on it, e.g. Python (Django) instead of PHP, Golang instead of Apache and PHP, PostgreSQL instead of MySQL.
  2. Normal MEAN stack, or a variant on it, e.g. DynamoDB instead of MongoDB, Ember.js instead of Angular.js.
  3. Serverless, use AWS Lambda to execute Node.js or Java code while running Angular/Ember.js on the client-side.

The LAMP stack
The traditional way of making a website would be through an open-source, free "LAMP" solution stack: https://en.wikipedia.org/wiki/LAMP_(software_bundle)

LAMP stands for:
  • Linux - the operating system (OS) of the server.
  • Apache - the web server running on the OS.
  • MySQL - the relational database the web server/web app connects to for its information
  • PHP - the web application / programming language that builds/displays the website, running on the web server.
You can substitute any part for an alternative if desired. E.g.
  • Windows instead of Linux (WAMP stack).
  • A Python server instead of Apache.
  • Oracle or PostgreSQL instead of MySQL.
  • Python or Perl instead of PHP.
You can get automatic installers to install the whole stack together, or just install them each separately and connect them up without much effort.
So all this would be running on a server, either physical or virtual.
LAMP is widely adopted throughout the world, a recognized "standard". But this has become less favourable in recent times due to the emergence of the MEAN stack. Problems include things like data transformation, supporting 3 different languages, scaling, and peak-demand performance.

The MEAN stack
Should Node.js be the mandatory choice? No.

MEAN is conceptually very similar to LAMP, but was newly coined in 2013: https://en.wikipedia.org/wiki/MEAN_(software_bundle)


MEAN stands for:
  • MongoDB - the NoSQL database.
  • Express.js - the web application framework that runs on Node.js.
  • Angular.js - the client-side JavaScript framework in the browser.
  • Node.js - the server-side Javascript environment (V8 engine).
MEAN is becoming widely adopted now because many people have started using NoSQL databases instead of the typical relational databases. Why? For scalability, performance, space, etc. There are many benefits to going with a NoSQL db. MongoDB is one popular open-source NoSQL db available to freely use. Alternatives are Cassandra, REDIS, and Neo4J (Neo4J being by far the least popular) for particular use-cases. AWS also offers its own NoSQL database: DynamoDB.

Not only that, but people have moved away from using PHP (in LAMP). I have not used PHP myself, but after speaking to colleagues with extensive experience, they all complain about it. Words like "horrible", "awful", "a real pain" get used often. And this is from everyone I have asked about it, who have used it. Instead if you look to what developers are starting to use now, or want to use, it's all about Node.js. Node.js is modular, so you can use whatever components you want, and it handles everything really well on both the server-side where everything gets run, as well as on the client-side through Angular.js for further processing. Even though it is relatively new, it has all the features needed to make it a great system to develop on.

Is Node.js overrated? Maybe. It has asynchronous non-blocking I/O! But so does everything else. Node as a server is cool! But so is everything else. Node has NPN libraries available! But so does everything else have its equivalent. Node has great performance! But Node is only single-threaded, not multi-threaded like everything else.

The Meteor framework
An extension to the MEAN stack is through the Meteor framework. It makes handling of Node.js much easier, as well as the connections with MongoDB and on the client with Angular.js.

Apparently it takes a lot of the complexity away because normally each part of the stack needs to be configured individually, which can be a nightmare to get going when you deal with so much different technology. Meteor takes care of this so it's really good for a solo developer to take care of everything in a short space of time. A good review, and link to a good video for its merits can be found: https://wiki.dandascalescu.com/essays/meteor_js_vs_the_mean_stack
This will need some investigation but perhaps using a Meteor framework would be preferable over a manual MEAN stack, there appear to many advantages in our situation, and for future scaling.

The Ember JS framework
Another JavaScript framework is Ember.js which has become very popular is recent times. It's apparently great for creating really snappy web applications for great performance and it's become an industry standard just like Angular.js from the original MEAN stack. This is essentially an alternative to using Angular.js, and there are strong augments in favour of Ember.js, or vice-versa.


The Django framework
Django is a Python framework for the web applications, essentially on the server-side. It would run on an Apache web server, and be connected to a relational database. The main aim is to enable better performance and productivity for creating new websites. Esentially this is would be replacing the "P" in LAMP.

The Ruby on Rails (Rails) framework
Finally, another common framework available is Ruby on Rails, or simply "Rails". It uses the Ruby programming language and it will need a web application like Phusion Passenger and Apache or Nginx, connecting to a database. Rails takes various design patterns to heart in order to reduce load on the developer, to make their life easier for dealing with complexity. Again, this would replacing the "P" in LAMP.

The Golang language and related framework
It's possible to use a web application stack using Golang (Go, from Google) as the programming language, optionally also using a Go web framework like Gorilla or GoCraft and a (Go) web application to run the server-side code on. Even then, a framework isn't super needed because Go has lots of good important standard libraries available out of the box without a need for an extra web framework.  Go can also work alongside other frameworks, like Angular.js. Go has libraries to connect to all AWS services too. The downside to choosing Go would be that it's an emerging/niche language since it's not so popular compared to Node or JS in general. So getting someone in might be a bit more difficult further down the line, but in reality, if you could find someone who has either experience or willing to learn/tackle Go, then you're almost certainly set with a very good programmer. Go has amazing performance (much better than Node.js), and you can even run your web server with a Raspberry Pi.

Go should be a serious contender, couldn't put it any better than this comment:
… if you want to do something like writing a tool, a server, a web application, a network application or an application to process, send/receive something or whatever that is called high-level. Things that previously people have to use such slow and unsafe scripting languages like Python, Ruby or PHP to write. You don't care about GC, you don't care about memory management, lifetimes and all that stuff, and you shouldn't care because Go is a compiled language which is more than fast enough to do ten times the work. You just want something simple, fast, compiles instantly and makes your code the easiest to read, to understand, to cooperate and to maintain. Choose Go, for the love of god choose Go because it's awesome for the job.

Big companies like Twitter are also using Go now, but big companies pretty much use a every technology somewhere.

Server-less architecture
The problem with both the LAMP and MEAN/Meteor stacks are that it requires you still to need to host/run/administer/backup/fix a virtual machine, web server, and database all by yourself. This is all in addition to the actual programming that needs to be done to write the web application. Another architecture type is available that eliminated the need for the virtual machine and web server: serverless.

Another, less popular way to host a website is directly through cloud services like AWS S3 cloud using Lambda functions . This eliminates the need for a web server, like how it could for (1) above. Here, static content would be hosted on S3, as well as dynamic JS. JS then gets executed on the client-side (browser), which then asks for further information through to Lambda, which will then query to the database or S3 storage by executing code (JavaScript or Java). Lambda would essentially replace the need for an web server running 24/7, and AWS would bill you according to how often Lambda was used (100ms execution timeframes).

For this type of implementation, the price should be a factor as well -t he cost of Lambda functions vs the cost of running, testing, administering an web server for the web app.

Not many people have adopted this style because people either aren't convinced it can scale (it can), or they would rather go for an industry recognised design/stack/architecture, or for cost reasons. I'm sure nobody got fired from a big company or a start-up for going with a LAMP stack, or a MEAN stack with Angular, or for using Ember. Telling your boss you want to go with a new serverless design that almost nobody else does in order to decrease costs and increase performance plus scalability? That might not go over too well, purely because it's seen to be riskier and unproven since not many people decided to do it.
…it’s a complete package for building a real, scalable backend service and fronting it with both mobile clients and a website, all without the need for servers or other infrastructure in any part of the system: frontend, backend, API, deployment, or testing. Go serverless!

Serverless Microservices: From <https://www.youtube.com/watch?v=w14NJkV5yAg>
Example implementations:



Side Notes


For the MEAN stack, the MongoDB would have to be hosted somehow, just like the web application has to be. MongoDB has a free open-source licence, but like anything apart from AWS Lambda, it would need to be hosted somewhere. MongoDB needs multiple nodes (instances) to operate effectively. Having to manage and backup this manually might be a real pain and cause a lot of extra effort for not much gain, e.g. primary/secondary setup, problems if nodes go down, problems scaling and adding more nodes, etc. A hosted solution might be less of a headache than self-hosting.

Wednesday, 13 April 2016

What are AWS Cloud Design Patterns?

The AWS Cloud Design Patterns (CDP) are a collection of solutions and design ideas for using AWS cloud technology to solve common systems design problems. To create the CDPs, we reviewed many designs created by various cloud architects, categorized them by the type of problem they addressed, and then created generic design patterns based on those specific solutions. Some of these problems could also be addressed using traditional data-center technology, but we have included cloud solutions for these problems because of the lower cost and greater flexibility of a cloud-based solution.
http://en.clouddesignpattern.org/index.php/Main_Page