More things for using K8s for real

In my current job, the team are working on deploying an application onto Azure using Kubernetes, so it was time to do some reading around the technologies that are being used.

Docker is used by the company as the universal packaging mechanism. The engine part of the application is written in C++, with a combination of Go and Python being used to wrap this to produce the final product. Multi-stage docker builds are used to build the application itself, as well as docker containers for running the various tests. The cloud version of the system is also deployed using Kubernetes  using more containers.

Docker Up & Running: Shipping Reliable Containers In Production by Sean Kane and Karl Matthias

This book was a great introduction to docker, with the first chapters providing a good introduction as to why containers are a good idea. It covers the common use cases, and goes through the many command line calls you will need to use to understand what is going on. The next chapter is also very good, with a good explanation of what is happening at the Linux level. There is also a chapter on docker-compose, which is interesting though not particularly relevant if you are deploying to Kubernetes.

Terraform Up & Running by  Yevgeniy Brikman

This book was also a good read. Terraform is used by the system at work to do the initial provisioning of the AKS cluster and associated networking and external resources. This short book gives as good overview of Terraform, and has a good running example that shows you how it is used for real.

The Master Algorithm: How the Quest for the ultimate learning machine will remake our world by Pedro Domingos

This was an easy read, which starts with a good overview to how machine learning has come of age. The book then describes the 5 tribes of machine learning – the symbolists, the connectionists, the evolutionaries, the Bayesians, and the analogisers, and gives a brief overview of how their techniques work. The rest of the book discusses how these techniques will need to be merged to get to a master algorithm, and the author covers some of the work he has done to do this on the Alchemy system.

Advertisements

And some more books

The Go Programming language by Alan Donovan and Brian Kernighan

I read this as preparation for a new job. It feels like the definitive guide to the Go language. The book is a great learning tool, highlighting practical uses of the language as it covers the syntax, though the language itself will definitely take some getting used to.

Clean Architecture by Robert C Martin

I’m not 100% sure of how I felt about this book. It mentioned lots of ideas that I had seen before – for example the SOLID principles, and linked them into ideas around components.  There is also a 45 year journey through the author’s career which was an interesting read, but I don’t think it helped much with the concepts.

I’ve also been watching a lot of videos on Linux and containers. This talk on the difference between containers and hypervisors is really good. The talk mentions technologies such as gvisor which is also covered in this blog post.

Some odds and ends

My time at Redgate is coming to an end, so I’ve been reading a fair few books on some of the new technologies that I’ll be using in my new job. More on that later.

First some interesting videos and posts from the last month or so.

I’ve been getting to grips with Comonads as part of my push to understand Haskell in more depth. This talk on implementing the Game of Life using Comonads was interesting,  and it also led me to watching this slightly more in depth talk. This talk on UIs being Comonads, part of a series on Purescript was how I started this path.

This article on KVM made sense, and gives enough of an overview to help you understand where KVM sits.

I’ve been using React a little, and have found it a great technology. Of course, I’d like to understand more about what is happening under the covers, and this series of blog posts  helped me understand the reconciliation process between the React tree and rendered browser DOM, by way of the new Fiber engine. This YouTube video is also a good overview.

MySQL High Availability

I found this book in the local library and thought I’d give it a read. I read the older version, which was written just after MySQL had extended the statement level replication to support row based replication. I enjoyed the level of detail, and the hands-on nature which shows you the commands you’ll need to run to start the replication process. The idea of using replicas so that writes go to the master and reads go to one or more slaves is an idea that seems to come up all the time in system design examples. This book gave me a good feel for how this all works. After I read this book, I found this blog post which gives a grief summary of the ideas.

Fluent Python

I have done bits and pieces in Python in the past, but wanted to get a better grip of the more advanced features of the language. I was lucky enough to find this book in a local bookshop.

The book has several sections:

• data structures, which talks about the basic sequence types, as well as dictionaries, sets and text
• functions as objects, which talks about functions as first class entities, and includes decorators and closures
• object oriented idioms which discusses object references, mutability and recycling, the meaning of a Pythonic object, interfaces and inheritance.
• control flow, which covers iterables, iterators and generators, contexts, coroutines and concurrency. This is a part of Python about which I knew nothing, and the book covered it really well.
• metaprogramming, which talks about adding dynamic attributes and properties to classes as well as class level metaprogramming

I really enjoyed the book. It discussed everything at the right level, with enough details to really understand what is happening under the covers. In fact, the author shows the byte code at times to help you understand what is going on.

Managing Kubernetes

This book talks about the operational side of managed a Kubernetes installation. As such you get to see a little more about what happens in the standard components, and it also covers monitoring, disaster recovery and authentication which are aspects you really need to understand if your business depends on it.

Pro .NET Memory Management

It’s impossible to describe how good this book is. It covers the implementation of the .NET garbage collector in great detail, often linking back to the sources. It works through various case studies to show you issues that you may need to work around, and it spends a lot of time showing you how to use ETW and perfview to diagnose issues in your application that are caused by garbage collection. It’s a massive book that takes quite a while to read, and I’m going to need to re-read it loads of times to really understand it all.

The level of detail in the book is truly awe-inspiring.

Quantum Space

I’ve always been interested int he unified theories of physics and wanted to understand the progress in Loop Quantum Gravity. This book covers some of the history of the theory and is a good read.

I’m hooked

It always amazes me how simple ideas can transfer from one domain to another.

A while ago I was looking at Azure Durable Functions. In this domain you want to write a function that can call out to another service, but for scalability reasons we’d like to avoid waiting for the result. Instead we’d like to stop the current execution and then reinstate the context when the value is available. Of course the difficulty is that this sounds like special compiler logic (like the type of transform used to implement generators in many languages). Instead, in the case of Azure Durable Functions, the designers demand that the functions are deterministic, and the trick is to store the previous results in an event log. We can then get to the same execution point by replaying the function – instead of making real outgoing calls, the system returns the previously logged return value at each point where the code tries to call out of the context, knowing that this will lead us down the same path that we executed the first time. This lets us wind forward to the same place and then execute the next step.

React has always had two ways of defining a component. You can use a class and override the various lifecycle methods (like componentDidMount) or you can write more functional (Pure) components that just return the mark up that they want to render. These functional components can be a lot easier to read, and it can be easier to share logic as it is easy for functions to call other functions. Adding state into such functional components is harder though, and the React team have introduced a concept called Hooks to do this. This again requires the idea of a function where outgoing calls are made in a  deterministic order, and this lets the system use this order to return different values.

There’s a good post that explains Hooks here, but the quickest introduction is a talk at React Conf with an informative excerpt here. In the example that they show in the talks, the functional approach makes the code much easier to understand than the class based approach – the latter can lead to related logic being spread across the class’s methods whereas the functional approach puts the code together.

While we are on the subject of React, there’s a post here talking about Flutter as an alternative to React Native.

Of CORSe, it’s easy to test

I read this informative post on CORS at the weekend, and realised that the best way to get to grips with it, is to try some experiments. I hadn’t realised before how easy this would be in C#. It’s easy to write a mini-web server that handles a single call using code like this.

            IPAddress ipAddress = IPAddress.Parse("127.0.0.1");
            TcpListener listener = new TcpListener(ipAddress, 8182);
            listener.Start();

            using (var clt = listener.AcceptTcpClient())
            using (NetworkStream ns = clt.GetStream())
            using (StreamReader sr = new StreamReader(ns))
            using (StreamWriter sw = new StreamWriter(ns))
            {
                var msg = sr.ReadLine();
                Console.WriteLine(msg);

                if (msg.StartsWith("GET"))
                {
                    string line;
                    while ((line = sr.ReadLine()) != "")
                        Console.WriteLine(line);

                    Console.WriteLine();
                    Console.WriteLine();

                    sw.WriteLine("HTTP/1.1 200 OK");
                    //sw.WriteLine("Access-Control-Allow-Origin: chrome://newtab");
                    sw.WriteLine();
                    sw.WriteLine("{a:2020}");
                }
            }

With that code running, you can start Chrome in Incognito mode, and then run the following code in the console.

var h = new XMLHttpRequest()
h.open("GET", "http://localhost:8182", true)
h.send()

which gives the error

(index):1 Access to XMLHttpRequest at 'http://localhost:8182/' from origin 'chrome://newtab' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Uncommenting the Access Control line in the above code allows the request to succeed.

Some more reading

There are three other books that I have recently read without writing up here.

React in action by  Mark Tielens Thomas, which was a good introductory book on React. It explains the various concepts and takes you through a number of examples in the first two parts of the book. The third part of the book talks about higher level architecture, covering Redux, server side rendering and there is also a chapter on React Native. I found it a useful informative read.

Redux in action by Marc Garreau, which talks about the Redux state management approach which integrates well with the React architecture (despite not exactly following the Flux architecture). Most of the book covers the use of Redux with React, though it can be used as a standalone component.

Functional Programming: Application and Implementation by Peter Henderson. I read this book while working in industry for a year before going to university (so 1984), and it blew my mind. It talks about why functional programming is such a powerful model, and defines a small Lisp language (LispKit Lisp) which it uses in subsequent chapters. It walks through writing an interpreter for Lisp in Lisp, and then goes on describe an abstract machine which we can compile the language down to, a variant of Landin’s SECD machine. The book describes how easy it is to write a compiler in Lisp for the Lisp variant. The book then looks at extensions to the semantics such as delayed evaluated (using lambda and thunks). Towards the end of the book, the author describes how to write a runtime to support LispKit and also gives the abstract machine code for the compiler itself. On GitHub you can find various implements, like this one in F#.

It was great re-reading the book, and I owe it a lot. Functional programming and Lisp spanned the first twenty years of my career, and it was this book that got me started. I taught myself C in order to implement a Lisp interpreter so I could play around, and various papers on OS implementation using LispKit got me really interested in the field.

 

 

 

Reliable, Scalable and Maintainable Systems

Designing Data-Intensive Applications: The big ideas behind reliable, scalable and maintainable systems by Martin Kleppmann

I was lucky to have a six week sabbatical over the summer, and felt that it would be a good time to read up on the technologies behind some of the large scale distributed systems that are around at the moment. This book is a great read for getting up to speed.

It has three sections. The first is on the foundations of data systems, and starts with a quick discussion of what the words reliability, scalability and maintainability actually mean. The book then moves on to the various data models, where the author discusses the birth of NoSQL , query languages and the various graph databases. The underlying implementations are covered, including B-trees, SSTables and LSM-trees,  and various indexing structures. The section finishes with a discussion of data encoding and evolution.

The second section covers distributed data, and there are chapters on replication, partitioning and the rather slippery notion of a transaction. Distributed systems can fail in many interesting ways, all covered in the next chapter, including some discussion of Byzantine faults. The final chapter in the section talks about consistency and consensus. In all of the discussion the author is really happy to go into low level implementation details, and all of the chapters have lists of references of papers that you can consult for more information.

The final section is on derived data – how do we process the mass of data that we have accumulated. The first chapter is on batch processing, which covers map-reduce and later variants. This is followed by a chapter on stream processing. The final chapter of the book is the author’s idea for the future.

This book is a great read. It goes into loads of implementation details which helps the reader really get to grips with the ideas, though it might take more than a single read to understand the many ideas that are covered.