Tuesday, 28 June 2016
Preparing Your Code for JDK 9
Preparing Your Code for JDK 9: Erik Costlow discusses what to consider when upgrading to Java 9, where to find early access releases and how to analyze library dependencies for unintentional reliance internal APIs.
Thursday, 23 June 2016
Scrum and Kanban
The DevOps paradigm can be implemented through many different methodologies. The 2 most popular are: Scrum and Kanban.
Scrum
In essence, this is Scrum:
Everything that needs to be done is part of a product backlog. These tasks are then put into the Spring Backlog, and a spring is like a month long. Then anything blocking is reviewed daily for a "stand-up" meeting, and everything should be delivered incrementally with a view to the end of the spring. Then this process starts over again.
You're supposed to have certain roles to ensure this runs effectively: a product master to manage the product/spring backlog, a sprint master so that the Scrum is being followed. Then the sprint backlogs are supposed to be estimated, people are supposed to take ownership of tasks, nothing is supposed to interrupt the sprint, stand-ups are supposed to be useful, multiple people are working on the same project, and tasks are done throughout the sprint so that "burn-down" and "velocity" can be shown in a chart.
In reality from my experiences nothing like that ever happens, there is a great potential for misuse of Scrum mandates, which are arguably out-of-date in today's world. Especially for a small project I would recommend implementing a Scrum-ban (hybrid, or Kanban methodology which is less restrictive, and has a greater allowance for testing in order to help ensure high code quality, instead of having such an importance of deliverables at the end of a sprint. In theory sprints are supposed to be a way for developers to be protected from immediate changes to requirements, but with the flexibility to make changes to your plans as you go along. But in reality a lot of time that doesn't happen that way, and development get interrupted, and medium/long-term vision of projects are not as highly regarded as they should be.
Kanban
It's a method for continuous improvement, that is you add things you want done to the "To Do" list, and prioritize them. Then when you're free, take a couple things to the "Doing" list. Set a conservative limit on how much work you can do at any one time. When they're done, but them across to another "Testing/Waiting" or "Done" list. When you've freed up a slot on the Done list, then pull something from the To Do list. If something bounces back into Doing from Testing, Waiting, or Done, then either move something back to To Do, repiroritise, or just carry on and deal with it, and only pull something new from To Do when you're free and back to a normal workflow. So it's supposed to be a way for "pulling" work instead of "pushing" work, e.g. work being pushed onto you constantly. Rather, you constantly pulling work onto yourself when you have the capacity to is much better. This way if things are taking too long, or if they are problems in the chain from To Do -> Done, then it should be quickly realised where bottlenecks are, and how to overcome them. This is something that Scrum normally fails at doing, what with its Sprints, Velocity, Poker Chips estimation, etc.
Build/Package/Test/Release/Deploy
To build/package/test/release/deploy, there are many software options out there in order to help streamline and automate the processes. There are tools available from Atlassian, AWS, and many others. Some of the most popular choices are listed below.
Atlassian
Bamboo - for both automated code deployment and continuous integration. ($10 for 5 users, one-time payment, self-hosted, or $10/month hosted).
SourceTree - standalone tool for visualising Git repos (free).
FishEye - search/monitor Git repos. ($10 for 5 users, one-time payment, self-hosted).
Crucible - for code reviews ($10 for 5 users, one-time payment, self-hosted).
AWS
CodeDeploy - for automated code deployment. There is no additional charge for code deployments to Amazon EC2 instances through AWS CodeDeploy.
CodePipeline - for continuous integration. One active pipeline for free each month.
Others
otherwise expensive.
AWS storage
AWS offers 3 main ways for data storage:
S3
Simple Storage Service (S3) - data is held in the cloud, in a "bucket". Inside a bucket, files and sub-directories can be created. The point of this type of storage is to write once, read may times perhaps from different services (EC2 instance, the Internet, etc). They can also be deleted easily. Data are automatically backed up and have a 99.9% uptime.
EBS
Elastic Block Storage (EBS) - these are volumes that can be added onto EC2 servers, think of them like hard drives. They can be scaled up/down, added, removed, backed-up as needed per instance.
Glacier
The Glacier is for remote "off-site" backups. Write once, read (almost) never. This should be for long-term archival of data, e.g. that was present in S3 for a long time, and will never normally be used again, save for restoration of backup data. Glacier prices are cheaper than S3 for storage, but have retrieval costs.
API deployment
There are build automation tools for programming languages, which can be integrated into IDEs, e.g. Maven for Java, and any other JVM-based language (Clojure, Groovy, Scala, Ivy). Maven also has excellent dependency management from the Maven Central repository (and any private repositories). For Python, there's PyPI. Perl has CPAN. Node.js has NPM.
Source Code Repositories
In ye olden days of yore, source code repositories were either CVS (Code Verification Store) or SVN (Subversion). Both were horrible at everything: usability, collaboration, stability, merging, committing, branching. Literally everything it was supposed to do. Then along came new technology that did it all properly: Git and Mercurial. Git has officially taken off throughout the world after everyone and their dog has realised how much it works amazingly well. I've used it a lot, and I'm more than happy with it. Git is open source and free, but again like anything, private repos would need to be paid-for hosted or self-hosted. Off-site hosting would be the best, that way you don't have to worry about the cost, time and effort of physical hardware or its administration.
The most common Git hosted options are:
GitHub
GitHub is the world's most popular hosted Git site. It is for a reason: works really well, has a great browser interface and offers an in-built wiki to explain the source code, and issues to track, organizations, user groups, profiles, etc. There is unlimited free allowance for public open-source repositories, so it's great for the scientific community. The main downside is that it's expensive for private use. You're looking at $25/month for a maximum of 10 private repositories (libraries). Eek.
BitBucket
BitBucket is the next most popular Git site. Much less so than GitHub, but it's still quite popular. Why? It works just as well as GitHub, but you can create an unlimited number of both public and private repositories. There is a free allowance for up to 5 users on a single private repository. After 5, then you have to pay. On the plus side, if you have an academic email address, if you add it to your BitBucket account (not even as your primary email address) then you automatically get upgraded to the academic features for unlimited everything (not for private Groups though, that should be implemented at the institute-level). This is a perk Atlassian offer in order to encourage, attract and benefit students and faculty. So that's right, BitBucket is made by good ol' Atlassian so it has its own issue tracker, wiki and integrates with the rest of the Atlassian world, i.e. JIRA and Confluence.
BitBucket can be self-hosted, again with a $10 one-off payment for 10 users.
CodeCommit
AWS's Git offering is CodeCommit. Someone else wrote up his review and I couldn't put it better myself:
Although I’m a fan of CodeCommits simplicity and automation support there are a few negatives to consider before migrating everything to it. There is a cost, albeit a small one, to using it. You’re charged by number of users accessing your repos and the number of git operations. I’m assuming I’m not going to hit the operations limit often but I will immediately be paying for an extra user or two. I’ll have my own AWS CodeCommit user for normal read/write repo access and I’ll need additional, locked down, users for my CI services. Access control is handled using IAM, this is both a positive and a negative depending on how much you like the JSON based policy language. There’s currently no concept of a public CodeCommit repo so you can’t start a code base off in private and then promote it to public view as you can with GitHub or BitBucket. The dashboard UI lacks the familiar collaboration features, there’s no pull requests, issue tracking or wiki built in. If you’re working in a team you’ll need a separate way to handle those. These current limitations means GitHub doesn’t need to be worried just yet, the lack of collaboration features alone will prevent most people from considering CodeCommit as an option even for private only repos.
Basically, it's just not worth it in a world where you have BitBucket and GitHub.
Managing technical projects: Documentation & Planning
Managing a technical
project means that all parts need to have proper oversight in order to encourage
and instill "agile" DevOps. This means you have to have the right
tools available to be able to track all the work that needs to be done, the
plans, sketches, architecture, design decisions, relationships, coupling,
issues, short/medium/long-term goals, timetables, etc. Literally anything to do
with planning and implementing a project. An example of that is this
documentation.
Tools come in 3
payment flavours: Free, Paid, and
Hosted.
- Often a tool will give you good or limited functionality for free, then ask you to pay for multiple user licences or to have all the features you want.
- This could be a one-time free, for a monthly/yearly subscription according to needs.
- Finally, in the case of more complex tools that need to be run on a server, they can instead be hosted for a monthly subscription fee.
E.g.
- Google accounts are free, and which gives you an allowance for email and document editing/storage to a certain limit. If you want more space, you need to pay a yearly subscription.
- JIRA can be purchased for a 1-time $10 fee for up to 10 user accounts, with 1 year of updates. There is no recurring charges, but no updates will be applied without a licence update. This would need to be hosted on your own server. Alternatively, JIRA can be hosted by Atlassian themselves for $10/month.
Documentation
What tools should be
used for documentation? Where should it be stored? How would it be accessed?
What are best practices to follow?
Currently my own
personal notes are written in OneNote, and Synced to OneDrive. It's free
to use (with a Microsoft account) on Windows, OSX, Android, iOS, Linux
(browser), and it's a very powerful note taking tool. For now it's fulfilling
my needs because I can privately share, edit and organise the pages very
easily, as well has a plethora of rich text formatting.
An alternative to
OneNote would to use Google Docs and save the documents to
Google Drive using a Google account. This is perhaps the next best thing, but
it lacks some flexibility when it comes to pasting in content wherever on the
page its needed, or having separate pages easily under one Section tab. This
would is free as well, with browser and mobile device tools, with paid-for business support available too.
Another alternative
is Evernote.
I have attempted to use Evernote, but I found it unintuitive for sharing notes.
Tagging is good, but sharing and collaborating seems less good. Sharing can be
done through messaging/Work Chat to invite others to view/edit/invite a note.
There is no organization, it has to be looked up through the Work Chat tab.
Searching doesn't work on shared Work Chat notes, and they can't be tagged
either. It's free, and there are paid accounts available too. Evernote also
lets you sort by creation/date edited/title, not by any custom order.
A more
"robust" system to use is Confluence. It's part of the Atlassian ecosphere. The main
benefit it has, apart from being very stable and usable within the web browser,
is that you have good control over user permissions, organizing spaces, and
integration with other good Atlassian products (e.g. JIRA, more on that later).
For small teams, pricing is $10/month hosted, for $10 one-off fee (to charity)
self-hosted (12 months of updates, 10 users). And then it goes up from there.
Planning
JIRA
Software from Atlassian is the #1 software tool used to manage
software projects. That means it's maybe the best tool for to handle this,
otherwise it wouldn't be able to make such a claim. It provides useful
functionality to track programming projects, issues, tasks, milestones,
sprints, TODOs, testing, deployment, etc. I.e., help to perform agile DevOps.
This can be also with SCRUM boards, or Kanban boards too, and various project
planning charts, diagrams, etc. It can also
be integrated really well into other Atlassian products, e.g. BitBucket. JIRA
Core also is included which has "normal" core functionality, i.e. for
non-software developers. Licence fees are the same as for Confluence for small
teams: $10/month hosted, for $10 one-off fee (to charity) self-hosted (12
months of updates, 10 users).
Trello
is a free board web application to add lists, cards, and information. It's very
flexibly with lots of features to add information everything. The free version
offers unlimited usage for the normal features, but it does not integrate into
anything, e.g. GitHub issues. Such features are behind the paid service. It's
very easy to use, and the only real downside is that lack of integration so
resources cannot be referenced easily/directly without a lot of manual
intervention or tracking.
Self-hosting options
Interested in self-hosting? Here are a load of good choices to explore:
- https://github.com/Kickball/awesome-selfhosted
- https://selfhosted.libhunt.com/
- https://www.reddit.com/r/selfhosted
Subscribe to:
Posts (Atom)