Kubernetes course has been expanded!

I’ve recently added a major update to the Kubernetes Microservices course. The original version of the course was a project based, hands-on approach to learning Kubernetes – which I think is by far the best way to learn. The problem with that is that it’s not always possible to fit every possible topic into a sample project – for example, Fleetman didn’t need any access control, so I didn’t bother with RBAC on the original course.

So this update fixes this! You can get the full course from Udemy or if you prefer to also get access to a curated, focused library of Java-stack related training, from VirtualPairProgrammers.com.

What’s new?

  • Resources and Limits – helps you to help Kubernetes manage the cluster effectively
  • Metrics – use the top command to monitor the resource usage of your pods and nodes
  • Horizontal Pod Autoscaling – let Kubernetes automatically scale up your deployments to handle spikes in load
  • Readiness and Liveness probes – make changes to your cluster without downtime
  • QoS and Eviction – understand why particular pods are evicted if a node runs out of resources
  • RBAC – use X509 certificates to restrict access to your cluster
  • ConfigMaps and Secrets – learn how to externalize configuration into a ConfigMap
  • Ingress Controllers – define routing rules so that multiple services can be accessed by a single point of entry – save money on LoadBalancers!
  • StatefulSets – by far the most confusing topic in Kubernetes, I hope I’ve demystified what they are (and aren’t! You don’t need them for persistence!)

But the biggest feature request for the update – Continuous Deployment! I spent ages building a CI/CD system which you can work with – it’s functional but just basic enough so that it isn’t overwhelming.


All Things Java Podcast 15 (plus a new course)

I don’t want the blog to be just “look at me! I’ve got a new course!!” – but the problem is when I’m writing a course, 100% of my attention is on that course. For the last one, I spent more hours than I dare admit trying to work out the implementation of HashAggegration in Apache Spark. While I’m doing that, I can’t think about anything else (too much info warning: look away now: even basic operations such as eating, cleaning and basic hygiene go out of the window. Weep not for my sacrifices to the community, and buy buy buy!)

So instead of just shoving up a “new course announcement!” I’ll lead this blog post by announcing I got together with Matt for an off the cuff podcast, and it’s just released here:

Sales spiel : we’re talking a lot on that podcast about the latest release (Apache Spark). You can get it here for (as far as I now, Udemy have odd sales) the lowest possible price: https://www.udemy.com/apache-spark-for-java-developers/?couponCode=SPARKBLOG

The podcast is an odd beast, we called it “All Things Java” but it’s a misnomer. Actually the design of the podcast is that it’s supposed to be the Virtual Pair Programmers team meeting – ie something we would have to do anyway – and we release it with the minimum of editing. Maybe if we mention any specific financials we might edit that out. Otherwise, it’s warts and all. We don’t pretend to be on top of the latest developments in Java, we don’t pretend to be master business people.

So I feel that the podcast might disappoint those wanting a pure Java podcast (I really miss the old “Java Posse” podcast, we certainly couldn’t replicate what they did), but I hope it might be of interest to anyone with an interest in Java (natch), but also business, content creation and the general problems of trying to be successful in software. Maybe we’ll rename it one day, if we can be bothered.


New Course: Kubernetes Microservices

Second course to be published at Udemy: Kubernetes Microservices.

I’ve set up a coupon code for a great deal.

Unlike the previous Docker course, this one is independent of programming language (Angular 6 used on the front end)

– although of course I can’t help dropping in a few references to Java along the way.

It’s a 12 hour hands on project: scenario – developers have pushed a set of images for a microservice system. Your mission, should you choose to accept it: define a Kubenetes workload locally, then deploy it to the cloud. Build in a level of resilience and then set up a proper ElasticSearch basic logging and monitoring system.

At the end of the course you’ll use your monitoring system to discover the developers did a pretty bad job in one of their images!

It was fun to record, hope it will be fun to watch and more importantly, a great set of real world skills for you.

A version of the course will appear in around a week at virtualpairprogrammers too.


Docker: Hands On Available at Udemy

I’ve just released my first course on Udemy. It’s a combination of the two modules I’ve previously published at VirtualPairProgrammers.

If you’re already a subscriber at VirtualPairProgrammers don’t buy the course again – but if not this course is a great way to get some real hands-on experience of Docker in a production cloud environment. I’ve set up a discount code for Udemy, follow this link for a great deal.

Kubernetes next up in a week or so!

Docker Build Hanging?

TLDR; create a .dockerignore file to filter out directories which won’t form part of the built image.

Long version: working on the upcoming Kubernetes course, with a massive deadline looming over me [available at VirtualPairProgrammers by the end of May, Udemy soon afterwards], the last thing I need is a simple docker image build freezing up, apparently indefinitely!

A quick inspection of running processes with Procmon (I’m developing on Windows) showed a massive number of files being read and closed:

This just a small extract from the process log.

This [every file in the repository being visited] is normal behaviour of the docker image build process – I guess I’ve just been working on repositories with a relatively small number of files (Java projects). This being an Angular project, one of the folders is “/node_modules” which contains a masshoohive number of package modules – most of which aren’t actually used by the project but are there as local copies. This directory can be easily regenerated and isn’t considered part of the project’s actual source code. [edit to add, it’s the equivalent of a maven repository in Javaland. The .m2 directory is stored outside your project, so this isn’t a concern there].

Turns out, the /node-modules folder contained 33,335 files whilst the rest of the project contained just 64 files!

Routinely, we .gitingore the /node_modules folder, and of course it makes sense to ignore this directory for the purposes of Docker also.

Simply create a folder in the root of the project, .dockerignore:

$ cat .dockerignore

You might also consider adding .git to this ignore list.

Now my docker image build is taking a few seconds instead of several hours. Perhaps I might meet this deadline after all…

Problems with dockerHost in Maven?

A problem reported against my Docker Module 2 course:

This happens on Windows Toolbox (ie anything below Windows 10 Pro) and possibly Mac toolbox as well.

If your Maven build is failing (with the fabric8 plugin) you need to add additional configuration. On the Docker Toolbox command line type

docker-machine env

Observe the output: you’re looking for the entries for DOCKER_HOST and DOCKER_CERT_PATH. Add the values you find here into your Maven pom, something like this:

...images tag etc here

Of course don’t just copy and paste these values – check they match the output of docker-env.

AllThingsJava Podcast

I’m back on the AllThingsJava podcast again. I had a shocking hangover when recording it so forgive my vague rambling.

Covered on this episode:

  • News – Javascript App gets sued for trademark breach – could it happen to us?
  • News – Java 10 features
  • News – (actually old news but since the last podcast) JavaEE is dead, long live JakartaEE.
  • Matt is planning a new course on Kotlin, coming soon.
  • I’m planning a new course on Kubernetes, coming even sooner. I’ll blog about this new course shortly. I should have done Kubernetes a year ago, I feel somewhat left behind and there are other good courses out there – I’m not sure how I can be different so I’m working on an “angle” right now. Hope I come up with something worth having.
  • There was some talk about whether we should do more on Javascript front ends. I think we should and I’m working on shoring up my skills in this area.

New Podcast – Java vs Scala

A new podcast is now available over at AllThingsJava. I’m not on it, which makes it even better than usual. The guest is Jon Humble, an experienced Scala developer.

We first met him when he presented a very funny talk which pitted Java against Scala in a kind of battle. Neither won because the presenters were too nice to declare a winner, so it was left to the viewer to decide…

…unlike me, Jon knows what he’s talking about so check out the podcast, lots of Kafka and similar buzzword goodness in there.

Java 10 – Local Variable Type Inference

In a previous post I listed the features added to Java 10. Naturally in a six-month update, the list isn’t long and most of them are updates to the runtime (such as very welcome garbage collector updates or internal changes).

But the number 1 headline grabber is the addition of type inference for local variables, and it’s a biggie. Let’s take it for spin.

What is type inference?

You’re all familiar with the following ritual:

 Customer customer = new Customer("Alan");

Which, if you think about it, is a bit absurd. We have to tell the compiler twice what type of object we’re creating. There is a reasoning behind this – technically the left hand side is telling the compiler what type we want the reference to be stored as, whilst the right hand side is the concrete object we’re creating. Sometimes, the difference is important and relevant – when programming to interfaces.


AbstractCustomer customer = new CreditCustomer("Alan");

Here want to instantiate a specific type of customer, but for the rest of the code, we only want to call the methods that are common to all customers. Polymorphism, blah blah blah. We’re not interested in that today, I just wanted to point out there is a reason for the left-hand-side-right-hand-side apparent duplication.

Here’s the thing – for the most part the two sides of the equation are duplicated. I want a customer and it needs to be of type customer.

Don’t Repeat Yourself is a bad thing and it’s almost like traditional Java wants us to repeat ourselves!

Anyway, stop waffling and cut to the chase. From Java 10 onwards, we can now let the compiler guess – or infer the type that we want…

 var customer = new Customer("Alan");

“var” is one of those rare things in Java, a brand new keyword! Java will now guess (it’s not hard is it?) that the type we want is Customer.

That’s it.

For the rest of the post I’ll describe a more realistic use of var. I’m picking an example from Spark, but don’t worry if you haven’t used Spark before, the example is actually generic.

A Real World Example

As usual with these toy examples, things don’t seem very groundbreaking but I believe this feature will have a major and positive impact on code productivity. Consider this monster from my Spark Training Course:

result = totals.mapToPair(tuple -> new Tuple2<Long, String> (tuple._2, tuple._1 ));

Now, I should know what the return type from this method call is. Erm, well, I should but it’s a bit of a headache. I know that the return class is some kind of “RDD”, I think a JavaRDD. And I can guess from the <Long, String> that the resulting return type will also have the same generic. So, I nervously type into the IDE:

JavaRDD<Long, String> result = totals.mapToPair(tuple -> new Tuple2<Long, String> (tuple._2, tuple._1 ));

Ouch. No! I now have a compile error…the type on the Left Hand Side is wrong. So now we reach a ludicrous situation: the compiler knows what the type should be, and I don’t. And the compiler won’t let me proceed until I guess correctly. (Ok, what I mean is – until I work out the right type).

Often on the Spark course I advise the viewer to simply leave off the return type and then use Quick Fix for the IDE to add in what it thinks is the right type!

Using the “Create Local Variable” Quick Fix should add in the left-hand-side declaration automatically.

I only advise using Quick Fixes if you know what you’re doing – in this case I’m basically asking Eclipse to do a type inference for me. But it’s very hit and miss – in this case, it gets it wrong. Or sort of half right:

For some reason (I care not why) the generics on the left hand side haven’t been inferred so the compiler is still moaning. A second “quick” fix will rectify this…

And – hooray, we finally have the right answer – it seems I had forgotten that the type was actually “JavaPairRDD”:

 JavaPairRDD<Long, String> result = totals.mapToPair(tuple -> new Tuple2<Long, String> (tuple._2, tuple._1 ));

But what a performance! And obviously the compiler knew the right answer all along. It was just teasing us.

So, to the point – when working in real coding situations like this, it can be easy to be unsure of the correct type you need for the declaration – and actually knowing the right type isn’t particularly illuminating. So, in Java 10 I could have avoided all that mess and jumped straight to:

 var result = totals.mapToPair(tuple -> new Tuple2<Long, String> (tuple._2, tuple._1 ));

Much cleaner and I hope in the long run, simpler. Note that none of this destroys strong typing – the type of result is still the same as it was, and we can only call the methods defined in the JavaPairRDD class, exactly as before.

I hope these kind of modern features are going to come thick and fast to future Java versions and that it will prove a compelling reason to upgrade – only time will tell…

[A footnote that after a long discussion Java decided not to implement a var/val keyword pair as in Scala. val would define an immutable value – whilst valuable it was decided this would over-complicate type inference, and it is arguably not all that useful on local variables anyway. That’s a bit disappointing but it’s true that immutability is orthogonal to type inference. I hope that a future version of Java will introduce some strong language level support for immutability.]

Trying Java 10 in Eclipse

If you’re keen to try Java 10 (of course you are!) then you can download the JDK from Oracle at the usual place.

I’m still with Eclipse (for the time being at least), and to use Java 10 (at the time of writing), you need to be on the latest version (currently Oxygen.3 aka 4.7.3) and you need to install (via the Marketplace) a Java 10 plugin.

Java 10 on the Marketplace

After installing the plugin, restart Eclipse and then define a JDK (using the usual tortuous Window -> Preferences -> Java -> Installed JREs and navigate to the folder you downloaded the JDK to).If you haven’t installed the plugin, this will fail because eclipse won’t recognise the new structure and layout of the JDK.

Now you can create a new project – I haven’t changed my default compiler compliance level so I’m getting a promising looking message saying the compliance level is going to be changed for this project…

New Project Window – looks like the compliance level is going to be set properly!

And that’s it. You can now create a class and start using the new features – the next blog post will explore what I think is the best of them, Local Variable Type Inference.

[edit1: I’m far too Eclipse-centric, I do need to address this soon. I can’t see any sign of Java 10 support in NetBeans but JetBrains/IntelliJ as usual are well ahead of the curve, they announced support for Java 10 as far back as last November. ]

[edit2: JetBrains are running a live Java 10 Webinar hosted by Trisha Gee on Thursday, Apr 5 2018, 4:00 PM – 5:00 PM CEST, check it out if you can.]