API deployment

There are build automation tools for programming languages, which can be integrated into IDEs, e.g. Maven for Java, and any other JVM-based language (Clojure, Groovy, Scala, Ivy). Maven also has excellent dependency management from the Maven Central repository (and any private repositories). For Python, there's PyPI. Perl has CPAN. Node.js has NPM. 

Source Code Repositories

In ye olden days of yore, source code repositories were either CVS (Code Verification Store) or SVN (Subversion). Both were horrible at everything: usability, collaboration, stability, merging, committing, branching. Literally everything it was supposed to do. Then along came new technology that did it all properly: Git and Mercurial. Git has officially taken off throughout the world after everyone and their dog has realised how much it works amazingly well. I've used it a lot, and I'm more than happy with it. Git is open source and free, but again like anything, private repos would need to be paid-for hosted or self-hosted. Off-site hosting would be the best, that way you don't have to worry about the cost, time and effort of physical hardware or its administration. 

The most common Git hosted options are: 

GitHub 

GitHub is the world's most popular hosted Git site. It is for a reason: works really well, has a great browser interface and offers an in-built wiki to explain the source code, and issues to track, organizations, user groups, profiles, etc. There is unlimited free allowance for public open-source repositories, so it's great for the scientific community. The main downside is that it's expensive for private use. You're looking at $25/month for a maximum of 10 private repositories (libraries). Eek. 

BitBucket 

BitBucket is the next most popular Git site. Much less so than GitHub, but it's still quite popular. Why? It works just as well as GitHub, but you can create an unlimited number of both public and private repositories. There is a free allowance for up to 5 users on a single private repository. After 5, then you have to pay. On the plus side, if you have an academic email address, if you add it to your BitBucket account (not even as your primary email address) then you automatically get upgraded to the academic features for unlimited everything (not for private Groups though, that should be implemented at the institute-level). This is a perk Atlassian offer in order to encourage, attract and benefit students and faculty. So that's right, BitBucket is made by good ol' Atlassian so it has its own issue tracker, wiki and integrates with the rest of the Atlassian world, i.e. JIRA and Confluence. 

BitBucket can be self-hosted, again with a $10 one-off payment for 10 users. 

CodeCommit 

AWS's Git offering is CodeCommit. Someone else wrote up his review and I couldn't put it better myself: 

Although I’m a fan of CodeCommits simplicity and automation support there are a few negatives to consider before migrating everything to it. There is a cost, albeit a small one, to using it. You’re charged by number of users accessing your repos and the number of git operations. I’m assuming I’m not going to hit the operations limit often but I will immediately be paying for an extra user or two. I’ll have my own AWS CodeCommit user for normal read/write repo access and I’ll need additional, locked down, users for my CI services. Access control is handled using IAM, this is both a positive and a negative depending on how much you like the JSON based policy language. There’s currently no concept of a public CodeCommit repo so you can’t start a code base off in private and then promote it to public view as you can with GitHub or BitBucket. The dashboard UI lacks the familiar collaboration features, there’s no pull requests, issue tracking or wiki built in. If you’re working in a team you’ll need a separate way to handle those. These current limitations means GitHub doesn’t need to be worried just yet, the lack of collaboration features alone will prevent most people from considering CodeCommit as an option even for private only repos. 

From <http://www.unixdaemon.net/cloud/aws-codecommit-initial-thoughts/>  

Basically, it's just not worth it in a world where you have BitBucket and GitHub.

Managing technical projects: Documentation & Planning

Managing a technical project means that all parts need to have proper oversight in order to encourage and instill "agile" DevOps. This means you have to have the right tools available to be able to track all the work that needs to be done, the plans, sketches, architecture, design decisions, relationships, coupling, issues, short/medium/long-term goals, timetables, etc. Literally anything to do with planning and implementing a project. An example of that is this documentation.

Tools come in 3 payment flavours:  Free, Paid, and Hosted.

• Often a tool will give you good or limited functionality for free, then ask you to pay for multiple user licences or to have all the features you want.
• This could be a one-time free, for a monthly/yearly subscription according to needs.
• Finally, in the case of more complex tools that need to be run on a server, they can instead be hosted for a monthly subscription fee.

E.g.

1. Google accounts are free, and which gives you an allowance for email and document editing/storage to a certain limit. If you want more space, you need to pay a yearly subscription.
2. JIRA can be purchased for a 1-time $10 fee for up to 10 user accounts, with 1 year of updates. There is no recurring charges, but no updates will be applied without a licence update. This would need to be hosted on your own server. Alternatively, JIRA can be hosted by Atlassian themselves for $10/month.

Documentation

What tools should be used for documentation? Where should it be stored? How would it be accessed? What are best practices to follow?

Currently my own personal notes are written in OneNote, and Synced to OneDrive. It's free to use (with a Microsoft account) on Windows, OSX, Android, iOS, Linux (browser), and it's a very powerful note taking tool. For now it's fulfilling my needs because I can privately share, edit and organise the pages very easily, as well has a plethora of rich text formatting.

An alternative to OneNote would to use Google Docs and save the documents to Google Drive using a Google account. This is perhaps the next best thing, but it lacks some flexibility when it comes to pasting in content wherever on the page its needed, or having separate pages easily under one Section tab. This would is free as well, with browser and mobile device tools, with paid-for business support available too.

Another alternative is Evernote. I have attempted to use Evernote, but I found it unintuitive for sharing notes. Tagging is good, but sharing and collaborating seems less good. Sharing can be done through messaging/Work Chat to invite others to view/edit/invite a note. There is no organization, it has to be looked up through the Work Chat tab. Searching doesn't work on shared Work Chat notes, and they can't be tagged either. It's free, and there are paid accounts available too. Evernote also lets you sort by creation/date edited/title, not by any custom order.

A more "robust" system to use is Confluence. It's part of the Atlassian ecosphere. The main benefit it has, apart from being very stable and usable within the web browser, is that you have good control over user permissions, organizing spaces, and integration with other good Atlassian products (e.g. JIRA, more on that later). For small teams, pricing is $10/month hosted, for $10 one-off fee (to charity) self-hosted (12 months of updates, 10 users). And then it goes up from there.

There are lots of alternatives out there, free hosted, hosted: XWiki, etc.

Planning

JIRA Software from Atlassian is the #1 software tool used to manage software projects. That means it's maybe the best tool for to handle this, otherwise it wouldn't be able to make such a claim. It provides useful functionality to track programming projects, issues, tasks, milestones, sprints, TODOs, testing, deployment, etc. I.e., help to perform agile DevOps. This can be also with SCRUM boards, or Kanban boards too, and various project planning charts, diagrams, etc. It  can also be integrated really well into other Atlassian products, e.g. BitBucket. JIRA Core also is included which has "normal" core functionality, i.e. for non-software developers. Licence fees are the same as for Confluence for small teams: $10/month hosted, for $10 one-off fee (to charity) self-hosted (12 months of updates, 10 users).

Trello is a free board web application to add lists, cards, and information. It's very flexibly with lots of features to add information everything. The free version offers unlimited usage for the normal features, but it does not integrate into anything, e.g. GitHub issues. Such features are behind the paid service. It's very easy to use, and the only real downside is that lack of integration so resources cannot be referenced easily/directly without a lot of manual intervention or tracking.

There are dozens of other options out there too, e.g. RedMine, Wrike, Asana, TeamClerk, Nostromo, etc. 

Self-hosting options

Interested in self-hosting? Here are a load of good choices to explore:

• https://github.com/Kickball/awesome-selfhosted
• https://selfhosted.libhunt.com/
• https://www.reddit.com/r/selfhosted

DevOps maturity model

How mature are your DevOps practices?
http://www.ibm.com/developerworks/library/d-adoption-paths/#N100E6
https://developer.ibm.com/urbancode/docs/continuous-delivery-maturity-model/

https://commons.wikimedia.org/wiki/File:Characteristics_of_Capability_Maturity_Model.svg

DevOps and AWS

DevOps is the practice about Development and Operations. It consists of a typical 'toolchain' of many tools that together describe how you should work on a software project. The main parts are as follows:

• Code – Writing the source code with a programming language. E.g. Java, Python, etc. This performed using an IDE, e.g. Idea IntelliJ for Java.
• Manage – Work has to be managed somehow, i.e. short, medium, long-term aims, sprints, issues, tasks, stories, bugs, milestones, etc. This can be managed with tools, e.g. JIRA.
• Store – The source code needs to be stored somewhere, e.g. using a Git repo hosted on GitHub, BitBucket. Similarly, built libraries will need to be stored and available somewhere, e.g. Nexus.
• Build/Package & Test/Release – The software will need to be built. Changes made to a library/repository that is used by another might have consequences. Or simply work done is ready for QA testing before production deployment. There are tools available to help automate and manage all of this for building software automatically according to your needs, e.g. TravisCI, Jenkins, TeamCity.
• Deploy – Software that has been built will then need to be deployed somewhere, i.e. for a test , development, or production environment. These environments and builds should be monitored too. Tools can be setup to help manage this as well.

AWS has solutions for all of these aspects, much like it does so for the infrastructure side of things. Examples of these are:

• CodeCommit - a private Git repo for your projects.
• Elastic Beanstalk for deploying, provisioning and testing servers.
• OpsWorks- manage stack implementations
• CodeDeploy - for deploying built software to AWS.

There is no obligation to use any of AWS' services, and you can use as many or as few of them as needed. They are paid-for services, but they all tend to have free limited usage still either during the first 12 months or over time, so you'll only pay for how much you go over the free provision, or plan to use in terms of numbers of commits, builds, etc. If you're considering to use AWS for these features, each service should be considered so that a good, timely decision is made so progress is not impeded and productivity is maintained.

DevOps is similar to being 'agile', i.e. employing methods and tools to increasing quality and quantity of work. Agile is more of a way of thinking, DevOps actually requires organizational changes - actual methods in the working culture and for delivering software.

DevOps can be implemented from the ground up to avoid future upheaval. It's really about applying agile principles, and there is no reason not to do this.

I'm sure we would all agree that it would be best to follow industry 'best practices' for tools and solutions. If other people have already performed analysis, and have detailed what options we have, and what the best options are for each use-case, then there is no reason to reinvent the wheel or procrastinate on matters. What matters most is finding a good solution quickly, and going with it. Even though this might not be the most perfect solution, perfect is the enemy of good.

An example of this would be for choosing a cloud platform: Google Cloud vs AWS. Both have almost identical offerings. Does it really matter which provider to go with? In general terms, no, as long as you go with something that fits what you want to do now, with a capacity for the future in mind. Such a decision does need to be agonised over in order to make rapid progress.

Website solution stacks

In summary, the normal options about how to create a website on a platform are:

1. Traditional LAMP stack, or a variant on it, e.g. Python (Django) instead of PHP, Golang instead of Apache and PHP, PostgreSQL instead of MySQL.
2. Normal MEAN stack, or a variant on it, e.g. DynamoDB instead of MongoDB, Ember.js instead of Angular.js.
3. Serverless, use AWS Lambda to execute Node.js or Java code while running Angular/Ember.js on the client-side.

The LAMP stack

The traditional way of making a website would be through an open-source, free "LAMP" solution stack: https://en.wikipedia.org/wiki/LAMP_(software_bundle)

LAMP stands for:

• Linux - the operating system (OS) of the server.
• Apache - the web server running on the OS.
• MySQL - the relational database the web server/web app connects to for its information
• PHP - the web application / programming language that builds/displays the website, running on the web server.

You can substitute any part for an alternative if desired. E.g.

• Windows instead of Linux (WAMP stack).
• A Python server instead of Apache.
• Oracle or PostgreSQL instead of MySQL.
• Python or Perl instead of PHP.

You can get automatic installers to install the whole stack together, or just install them each separately and connect them up without much effort.

So all this would be running on a server, either physical or virtual.

LAMP is widely adopted throughout the world, a recognized "standard". But this has become less favourable in recent times due to the emergence of the MEAN stack. Problems include things like data transformation, supporting 3 different languages, scaling, and peak-demand performance.

The MEAN stack

Should Node.js be the mandatory choice? No.

MEAN is conceptually very similar to LAMP, but was newly coined in 2013: https://en.wikipedia.org/wiki/MEAN_(software_bundle)

MEAN stands for:

• MongoDB - the NoSQL database.
• Express.js - the web application framework that runs on Node.js.
• Angular.js - the client-side JavaScript framework in the browser.
• Node.js - the server-side Javascript environment (V8 engine).

MEAN is becoming widely adopted now because many people have started using NoSQL databases instead of the typical relational databases. Why? For scalability, performance, space, etc. There are many benefits to going with a NoSQL db. MongoDB is one popular open-source NoSQL db available to freely use. Alternatives are Cassandra, REDIS, and Neo4J (Neo4J being by far the least popular) for particular use-cases. AWS also offers its own NoSQL database: DynamoDB.

Not only that, but people have moved away from using PHP (in LAMP). I have not used PHP myself, but after speaking to colleagues with extensive experience, they all complain about it. Words like "horrible", "awful", "a real pain" get used often. And this is from everyone I have asked about it, who have used it. Instead if you look to what developers are starting to use now, or want to use, it's all about Node.js. Node.js is modular, so you can use whatever components you want, and it handles everything really well on both the server-side where everything gets run, as well as on the client-side through Angular.js for further processing. Even though it is relatively new, it has all the features needed to make it a great system to develop on.

Is Node.js overrated? Maybe. It has asynchronous non-blocking I/O! But so does everything else. Node as a server is cool! But so is everything else. Node has NPN libraries available! But so does everything else have its equivalent. Node has great performance! But Node is only single-threaded, not multi-threaded like everything else.

The Meteor framework

An extension to the MEAN stack is through the Meteor framework. It makes handling of Node.js much easier, as well as the connections with MongoDB and on the client with Angular.js.

https://wiki.dandascalescu.com/essays/meteor_js_vs_the_mean_stack

Apparently it takes a lot of the complexity away because normally each part of the stack needs to be configured individually, which can be a nightmare to get going when you deal with so much different technology. Meteor takes care of this so it's really good for a solo developer to take care of everything in a short space of time. A good review, and link to a good video for its merits can be found: https://wiki.dandascalescu.com/essays/meteor_js_vs_the_mean_stack

This will need some investigation but perhaps using a Meteor framework would be preferable over a manual MEAN stack, there appear to many advantages in our situation, and for future scaling.

The Ember JS framework

Another JavaScript framework is Ember.js which has become very popular is recent times. It's apparently great for creating really snappy web applications for great performance and it's become an industry standard just like Angular.js from the original MEAN stack. This is essentially an alternative to using Angular.js, and there are strong augments in favour of Ember.js, or vice-versa.

The Django framework

Django is a Python framework for the web applications, essentially on the server-side. It would run on an Apache web server, and be connected to a relational database. The main aim is to enable better performance and productivity for creating new websites. Esentially this is would be replacing the "P" in LAMP.

The Ruby on Rails (Rails) framework

Finally, another common framework available is Ruby on Rails, or simply "Rails". It uses the Ruby programming language and it will need a web application like Phusion Passenger and Apache or Nginx, connecting to a database. Rails takes various design patterns to heart in order to reduce load on the developer, to make their life easier for dealing with complexity. Again, this would replacing the "P" in LAMP.

The Golang language and related framework

It's possible to use a web application stack using Golang (Go, from Google) as the programming language, optionally also using a Go web framework like Gorilla or GoCraft and a (Go) web application to run the server-side code on. Even then, a framework isn't super needed because Go has lots of good important standard libraries available out of the box without a need for an extra web framework.  Go can also work alongside other frameworks, like Angular.js. Go has libraries to connect to all AWS services too. The downside to choosing Go would be that it's an emerging/niche language since it's not so popular compared to Node or JS in general. So getting someone in might be a bit more difficult further down the line, but in reality, if you could find someone who has either experience or willing to learn/tackle Go, then you're almost certainly set with a very good programmer. Go has amazing performance (much better than Node.js), and you can even run your web server with a Raspberry Pi.

Go should be a serious contender, couldn't put it any better than this comment:

… if you want to do something like writing a tool, a server, a web application, a network application or an application to process, send/receive something or whatever that is called high-level. Things that previously people have to use such slow and unsafe scripting languages like Python, Ruby or PHP to write. You don't care about GC, you don't care about memory management, lifetimes and all that stuff, and you shouldn't care because Go is a compiled language which is more than fast enough to do ten times the work. You just want something simple, fast, compiles instantly and makes your code the easiest to read, to understand, to cooperate and to maintain. Choose Go, for the love of god choose Go because it's awesome for the job.

From <https://www.reddit.com/r/golang/comments/29ua55/rust_or_go/ciojqcg>

Big companies like Twitter are also using Go now, but big companies pretty much use a every technology somewhere.

Server-less architecture

The problem with both the LAMP and MEAN/Meteor stacks are that it requires you still to need to host/run/administer/backup/fix a virtual machine, web server, and database all by yourself. This is all in addition to the actual programming that needs to be done to write the web application. Another architecture type is available that eliminated the need for the virtual machine and web server: serverless.

Another, less popular way to host a website is directly through cloud services like AWS S3 cloud using Lambda functions . This eliminates the need for a web server, like how it could for (1) above. Here, static content would be hosted on S3, as well as dynamic JS. JS then gets executed on the client-side (browser), which then asks for further information through to Lambda, which will then query to the database or S3 storage by executing code (JavaScript or Java). Lambda would essentially replace the need for an web server running 24/7, and AWS would bill you according to how often Lambda was used (100ms execution timeframes).

https://d0.awsstatic.com/whitepapers/AWS_Serverless_Multi-Tier_Architectures.pdf

For this type of implementation, the price should be a factor as well -t he cost of Lambda functions vs the cost of running, testing, administering an web server for the web app.

Not many people have adopted this style because people either aren't convinced it can scale (it can), or they would rather go for an industry recognised design/stack/architecture, or for cost reasons. I'm sure nobody got fired from a big company or a start-up for going with a LAMP stack, or a MEAN stack with Angular, or for using Ember. Telling your boss you want to go with a new serverless design that almost nobody else does in order to decrease costs and increase performance plus scalability? That might not go over too well, purely because it's seen to be riskier and unproven since not many people decided to do it.

…it’s a complete package for building a real, scalable backend service and fronting it with both mobile clients and a website, all without the need for servers or other infrastructure in any part of the system: frontend, backend, API, deployment, or testing. Go serverless!

From <https://aws.amazon.com/blogs/compute/microservices-without-the-servers/>

Serverless Microservices: From <https://www.youtube.com/watch?v=w14NJkV5yAg>

Example implementations:

• http://www.infoq.com/articles/mars-rover-application-DynamoDB
• http://highscalability.com/blog/2015/12/7/the-serverless-start-up-down-with-servers.html

Side Notes

For the MEAN stack, the MongoDB would have to be hosted somehow, just like the web application has to be. MongoDB has a free open-source licence, but like anything apart from AWS Lambda, it would need to be hosted somewhere. MongoDB needs multiple nodes (instances) to operate effectively. Having to manage and backup this manually might be a real pain and cause a lot of extra effort for not much gain, e.g. primary/secondary setup, problems if nodes go down, problems scaling and adding more nodes, etc. A hosted solution might be less of a headache than self-hosting.