And some more links

Things have calmed down a little at work, so it’s time to go through a list of links that I have accumulated over the last few months. Here are some of the interesting ones.

Preparing for a system design interview
Tracking in in-app browsers
When and why does a component re-render in React?
The .NET 7 preview 7 release, including links to Source Generators for P/Invoke and low level struct improvements
Some views on TDD which I agree with
Writing a hypervisor from scratch, a great tutorial
Scaling Git’s garbage collection
How to clone a running VM in seconds
An article on separation logic (for proving program properties when address aliasing can happen)
The infinite tower of meta-circular interpreters
20 dynamic programming interview questions
A simplified C++ memory model for x86
A note on LXC
Inspecting WebViews in MacOS
Database storage engines under the hood
An ongoing CLR issue with Ref locals
Stubs and mocks break encapsulation
Ten years of TypeScript
How Wine works
Support swap on Linux for Kubernetes
Benchmarking legacy x86 instructions

Advertisement

Inside Facebook, and API security

Two more books off the pile.

An Ugly Truth: Inside Facebook’s Battle for Domination by Sheera Frenkel and Cecilia Kang

This gives a journalist’s view of the fall of Facebook, from the darling of the tech industry to a company mired in controversy over various scandals. It’s an interesting read, and the many interviews allow the authors to give very in-depth details on the various scandals and how Facebook tried to handle them – from spin to political meetings of various kinds.

Advanced Api Security: OAuth 2.0 and Beyond by Prabath Siriwardena

This books covers OAuth 2 and its related profiles, which allow you to expose Enterprise Apis in a safe way so that they can be targeted by web applications, single page applications native mobile and browser-less applications. The author covers the whole subject area well with code that demonstrates some of the parts of the protocols and designs. There are also many good appendixes where the author covers the evolution of identity delegation on the web, as way as many other facets of this technology.

Some C# odds and ends

I’ve been taking some Microsoft Exams lately, but have finally had time to look through a long set of blog posts that I have accumulated. Amongst them were a few interesting posts related to C#, so I thought I’d list them here.

This is the design document for on-stack replacement, which has been added to the CLR in recent releases (and is still being improved). On-stack replacement has enabled better tier compilation – you start with a basic compilation of a given function, watch for how it is being used (for example, noticing the common paths through it), and then spent time compiling a more specific optimised version for the arguments that are being passed at runtime. This DPGO (dynamic profile guided optimisation) is great, and it is easy to make new invocations of a method use the more specific version, but if you have on stack instances that are, say, looping, you’d also like to transition them to using the new code. OSR lets you do this, changing the stack frames of executing versions of the old version of the code, to make them use the new version. This has enabled the CLR to use tiered compilation in more scenarios, and allowed methods to specialize for the data that the current run of the program is being passed (compared to PGO – profile guided optimization – which uses examples to specialize, though let’s hope you profiled using the right examples for your current run). It’s a great technology, and one that the JVM has had for a long time.

C# has emphasised low level performance and the avoidance of garbage collection in recent releases. In particular, stack allocated structs were added a while back. This proposal attempts to tidy up the ref struct and other added struct types, some of which are used in the implementation of Spans.

This post discusses the improvements in the C# library support of regular expressions and this post feels like a blast from the past as it discusses using COM from C#. While we are talking about runtimes, this post discusses how Go and the CLR differ, and this paper discusses some recent attempts to figure out how much garbage collection in a language costs you.

Some other good posts included: a list of the common machine learning algorithms

Distributed system distilled

Understanding Distributed Systems: What every developer should know about large distributed applications by Roberto Vitillo

This is, by far, the best book I have ever read on distributed systems. It’s about 250 pages of information starting with communication via HTTPS and discoverability, coordination via distributed notions of time, replication and distributed transactions and their non-transactional equivalents, scalability via functional decomposition and partitioning, resiliency via an analysis of failure modes, both upstream and downstream, and even has a final chapter on testing and operations which converts monitoring and alerting. It goes into detail about all of the items, putting together just the right level of information about the concepts – things like retry amplification are covered in a couple of paragraphs of great explanation, and ideas like chain replication where just explained brilliantly.

An absolutely fabulous book that every developer should read!

[I should also mention that if you are interested in Logic and application, there’s a great multi-hour episode of the Type Theory Forall podcast that talks about the context of Goedel’s incompleteness theorems and a lot more]

And yet more

Programming Microsoft Azure Service Fabric by Haishi Bai

I’ve been doing a lot of Kubernetes lately, and since that has a Google heritage from systems like Borg, I thought it was probably time to have another quick look at Service Fabric which was/is the base layer for many Microsoft services. Unsurprisingly, the concepts in Service Fabric are every much like those you find in other systems, though it felt to me that there was more of an emphasis on sharding and actor patterns – I think that some of these ideas have been taken across to Kubernetes with Microsoft’s DAPR. In Service Fabric the services can be stateful or stateless, and there are the usual mechanisms for rolling out changes across a cluster, concepts of scalability and performance, and availability and reliability.

The later chapters on Linux containers and advanced concepts (serverless and artificial intelligence) felt like standard expected things these days, but were probably innovative when the first version of the book was written.

Linux Kernel Programming: A Comprehensive guide to kernel internals, writing kernel modules and kernel synchronization by Kaiwan N Billimoria

This is a big thick book on the Linux kernel and how to write modules to plug into the kernel. As a way to write these kernel modules, it covers the memory management inside the kernel and the way it handles threads in great detail. For the former, you’ll get to understand the mappings of virtual memory and the page tables, on the way to understanding where your modules should allocate their memory. For the latter, you’ll get a great understanding of the Linux tasks and how they are scheduled, and the locks you will need to use to make your code race free.

The book is really practical, with instructions on how to set up virtual box to safely set up many of the kernel programming exercises.

I absolutely loved this book, and will be spending time working on the examples to increase my understanding.

Designing Microservices Platforms with NATS by Chanaka Fernando

We’ve been using NATS as a message bus at work in an application that I have inherited, so I wanted to learn a little more about it. This book covers the NATs application in its early chapters but also covers the ideas behind Microservice architectures too. It was an interesting read.

Another set of books

React Hooks In Action: With Suspense and Concurrent Mode by John Larsen

I’d been away from React for a while and noticed that React 18 has just been released. This reminded me of the talk about Concurrent mode (the ability for a render to be interrupted and then restarted if a higher priority task comes along) and Suspense (seamlessly handle data loading notification in the GUI) a while back when I last did React in my day job.

This book is a great guide to modern React. It discusses what hooks are and how they work, and then walks through an example application that uses a number of the standard React hooks. The application feels really realistic and the author explains the needs for the hooks and how to use them really well – useState, useReducer (part of the eternal question of how to handle component state across the application), useEffect and useRef. There is an initial chapter on managing application state which introduces useCallback and then useMemo.

This section then ends with chapters on managing state with the Context Api, creating your own hooks (by putting together existing hooks, and at the same time respecting the Rules of Hooks), and then a discussion of some of the available third party hooks (and React Query).

The shorter part 2 of the book covers Suspense and Concurrent Mode. The former is introduced in a chapter that covers lazy loading and error boundaries, and then there is a chapter on data fetching and how it interacts with Suspense. The last chapter looks at Concurrent Mode and the future of React.

The book is great – it is just the right length and covers the material in a way that answers most of the first level questions. Highly recommended!

Learning GraphQL – Declarative Data Fetching for Modern Web Apps by Eve Porcello and Alex Banks

Using REST to fetch data from a complicated data model feels increasingly like hard work for the client. There are often lots of times when you fetch an object and then need to issue requests for each of the children, wiring the navigation of the object graph by doing a fetch for each edge. It has often felt to me like a single request with more specifics about the navigation of the graph to the desired properties was the way to go and things like OData and GraphQL offer this.

This books is short introduction to GraphQL. It starts by describing where GraphQL fits in, with a discussion of the issues with REST. There’s a very brief introduction to graph theory and then a chapter on the GraphQL language – this covers fragments, mutations and subscriptions (GraphQL is not just for reading data but allows applications to write data, and also allows you to keep listening and be informed when the data changes).

There is a chapter on schemas, and then an example of how to implement a schema using Apollo Server, an open source GraphQL library. The book finishes with chapters on GraphQL clients (particularly Apollo Client) with some focus on their caches. The last chapter covers GraphQL in the real world, discussing things like timeouts and security, as well as community resources you can access.

The book is short, but provides all the information that you need.

Software Architecture: It’s hard!

Software Architecture: The Hard Parts – Modern Trade-off Analysis for Distributed Architectures by Neal Ford, Mark Richard, Pramod Sadalage and Zhamak Dehghani

This is a good book about the hard architectural decisions that a team face when breaking a monolith into a micro-services architecture. It covers a lot of very interesting topics like architectural fitness functions and is full of lots of practical advice on how to weigh up the pro and cons. I’m not sure I’m the greatest fan of the storyline which follows a team as it splits up a support application from monolith to microservices, but all of the material that hangs off the chapters of this story is really good. The book covers everything from choice of database type to service granularity to the different types of transactional sagas to choices between choreography and orchestration. It’s a great read.

Ceph in detail

Mastering Ceph: Infrastructure storage solutions with the latest Ceph release by Nick Fisk

We’ve started using Ceph, or more specifically Rook-Ceph as a storage solution for some Kubernetes work that we are doing. We started treating Ceph itself as a bit of a black box, but as we’ve moved more towards the performance side of things, it has become apparent that we need to have a better understanding of what is happening under the covers.

This book is a little old, but does give a good understanding of Ceph. There are general overview chapters about Ceph, and what its unique selling points are, and there are some Vagrant scripts that allow you to do a Ceph deployment yourself, in a couple of different scenarios. I loved the way that the book gets into the details by getting you to run various command line commands to understand what is going on. The second section of the book is about operating and tuning Ceph, and gave us many insights as to what was happening beneath the covers, and Section 3 on troubleshooting is going to come in useful as we deploy our K8s application to more customer sites.

For us, some parts of the book weren’t particularly relevant, as we won’t be using CephFS for example, but the book is definitely worth a read if you want to see across the whole of the Ceph system. We still have many questions about how Ceph is driven via Rook in the K8s world, but this was a great way to understand the storage architecture.

What are the chances of that?

I put together a lightning talk on Quantum Computing. I’ve recently got into Quantum Computing and find it a fascinating subject. There was, of course, too much material to fit in to the talk – I was aiming to try and cover all of the misconceptions about Quantum Computing, but in the end there was just too much to cover to make this really the subject of the talk.

Secure your containers

Container Security by Liz Rice

I really enjoyed this book which takes you through container security. It starts with a good introduction on container security threats, including security boundaries and the issue of multi-tenancy. After that, we look at some of the security mechanisms that Linux offers – systems calls and permissions including capabilities and features like setuid and setgid. This is followed by a good chapter on cgroups and how they can be used to control resource usage for a group of processes. The author then goes through the means that Docker uses to isolate its containers – the various Linux namespaces, the how the root is changed to isolate the container’s filesystem from the outside world.

There’s a great chapter on virtual machines, with a discussion of the difference between Type I and Type II VMs and how they work. This discusses how they differ from containers. This is followed by a chapter on container images – in particular what they contain in the various layers. We then look at what vulnerabilities images can contain, and how we can use scanning to detect them. Linux mechanisms to tighten container isolation are then covered – tools like Seccomp, AppArmour, SELinux and gVisor, and then moving away from containers to Kata Containers, FireCracker and Unikernels.

Chapters follow on breaking container isolation which includes running rootless containers, and the use of the –privileged flag. Then a chapter on container network security, one on TLS, how to pass secrets securely to containers, and container runtime protection using profiles.

This is quite a short book which is packed with loads of information. I really enjoyed the “build your own docker” chapters to build up knowledge about the way containers work, and the explanations are really great.