Continuing with my exploration of understanding physics from first math principles (see previous post on functionals), I wanted to learn more about Sobolev spaces.

These are a type of vector space named after the Soviet mathematician Sergei Lvovich Sobolev (1908-1989), featured on the thumbnail.

Motivation

Usually when we discuss differentiability, we have functions that are either differentiable or not.

In the real world however, we often need to handle functions that are not strictly differentiable everywhere but are almost there. A simple example is the function $f(x) = \abs{x}$ which is differentiable everywhere except at $x = 0$.

There’s a more relaxed version of differentiability called the weak differentiability which we’ll cover soon, but the idea is that a Sobolev space is the vector space of weakly differentiable functions.

Before we go there, let’s lay down some nomenclature.

Function Spaces

A vector space in which the elements are functions is generally called a function space. There’s a special notation for a select group of function spaces, which are associated with the properties of the functions that are part of it.

The set of continuous functions with domain $\Omega$ is denoted by $C(\Omega)$. We add a superscript $k$ indicating how many times these functions can be differentiated, so $C^1(\Omega)$ is the set of functions that have a continuous derivative. The set of smooth functions (infinitely differentiable) is denoted by $C^\infty(\Omega)$.

A special property we’ll need later is called compact support. This basically means that there exists some compact set $K \subset \Omega$ outside of which the function is 0. One example of such a function is:

These functions have the subscript $_c$, e.g. $C^\infty_c(\Omega)$.

The set of (Lebesgue) integrable functions is denoted by $L^p(\Omega)$. More formally:

A more relaxed version is that the function only needs to be integrable in compact sets. This is denoted by $L_{LOC}$:

Note that $L^p(\Omega) \subseteq L_{LOC}^p(\Omega)$. One example is $f(x) = 1$ for $\Omega = \mathbb{R}$. In this case compact set means a bounded closed interval $[a, b]$. We have:

So $f \in L_{LOC}^p(\Omega)$. But if the interval is unbounded, then the integral is not defined so $f \not \in L^p(\Omega)$.

We also have $W^{k, p}$ which is a vector space of the functions in $L^p$ such that all their weak derivatives up to order $k$ are also in $L^p$:

We haven’t defined weak derivatives yet though, but we’ll do so shortly. For now it suffices to say this vector space is what we call a Sobolev space.

Weak Derivatives

Let’s start with the formal definition. Let $\Omega$ be an open subset of $\mathbb{R}^n$ and $u \in L_{LOC}^1(\Omega)$. We say that $v \in L_{LOC}^1(\Omega)$ is a weak derivative of $u$ in the $i$-th direction (denoted as $v = \partial_i u$) if:

Let’s explain what each term means. The integral is over the infinitesimal box on $\mathbb{R}^n$ in the subspace $\Omega$. The function $\varphi(x)$ is called a test function and because it has a compact support, it’s only non-zero in that compact set, so we can interpret it as a bitmask in programming or a microscope that “focuses” a function $f$ when we multiply it by $\varphi$.

The notation $\partial_i$ is the partial derivative on the dimension $i$ (i.e. with respect to $x_i$) and is a shorthand for $\partial_i f = \partial f / \partial x_i$. A more compact notation if we omit the function argument:

Intuition

I still don’t have a good intuition behind why weak derivatives are useful. The least unsatisfying one is this: pointwise derivatives are too strict. The weak derivative averages out $u$ locally (in the compact support of $\varphi$) by multiplying it with $\partial_i \varphi$. Since $\varphi$ is smooth, it helps get rid of the kinks in $u$.

This reminds me of the use of convolution in image processing to smooth out pixels by taking the average of the surroundings.

The use of $\partial_i \varphi$ instead of just $\varphi$ is so that we get what would correspond to $\partial_i u$ on the other side of the equality.

Example

To get an idea on how to use this formulation, consider the function $f(x) = \abs{x}$ with $x \in \mathbb{R}$, for which we want to find the derivative. It’s not differentiable at $x = 0$ because of the “edge”.

For the rest of the domain we have $f’(x) = 1$ for $x \gt 0$ and $f’(x) = -1$ for $x \lt 0$. This is the function we’ll tentatively take as the function $v$. We just need to show it satisfies $(1)$. Note we didn’t specify what $f’(0)$ is like, but it can be anything. We’ll call this function $\mbox{sgn}(x)$.

Lemma 1. $\mbox{sgn}(x)$ is the weak derivative of $\abs{x}$

So in practice we do not need to list all the test functions. We just use the specific property that it has a compact support but otherwise don’t make any assumption about them.

Derivation

Let’s now understand where this formula comes from. Suppose for now that $u$ has a derivative. Since $\varphi$ is a smooth function, it also has a derivative. Suppose we multiply them and want to take the derivative on dimension $i$. We can use the product rule:

Here we’re omitting the $(x)$ parameter for simplicity. Now integrate over the entire domain $\Omega$:

We can think of $\partial_i$ as a gradient if we multiply by $e_i$ (the basis vector):

or in our case

so

By the divergence theorem, we have that

But because $\varphi$ has compact support, it means outside that region it is 0. This includes the boundary $\delta \Omega$, so each term on the righthand integral is $0$ and thus:

and:

and finally:

which is the exact form as $(1)$. The only thing is that we assumed that $u$ is differentiable. So instead of requiring that $u$ is differentiable, we just require a function $v$ to exist that satisfies $(1)$. For a differentiable function, $v$ coincides with $\partial_i u$.

The idea being “If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck”. We’ll look at this from a different angle next.

Representation, an analogy

Recall that in the post about functionals we covered the Riesz Representation Theorem which basically says that if $H$ is a Hilbert space then there exists some element in it, $y$, such that any linear functional $f$ on $H$ can be represented as the inner product $f(x) = \langle x, y\rangle$ for all $x$.

We have an analogous case for weak derivatives. For a fixed function $u$, define the functional:

we can show this functional is linear. So an analogous result to Riesz’s is that there exists a function $v$ such that $L(\varphi)$ can be represented as:

For all $\varphi$. The difference is that in Riesz it guarantees such a $y$ exists, but here $v$ only exists if $u$ has a weak derivative. Also for Hilbert spaces $y$ is from the same domain as $f(x)$, while here the domain of $L(\varphi)$ is not necessarily the same as $v$.

Ok, so now we know where the equation comes from and that it’s tied to integration by parts and test functions, but why choose this specific property among all possible properties? We’ll now cover examples to help us see why it’s useful.

Properties

In the example, we saw that $f = \mbox{sgn}$ is a weak derivative of $\abs{x}$ but that we can set any value for $f(0)$. At first glance this suggests that weak derivatives are not unique like derivatives are. But there’s a slightly weaker result that shows they’re the same almost everywhere (more formally, those with non-zero measure), as shown in Lemma 2:

Lemma 2. Let $v_1$ and $v_2$ be the weak derivative of $u$ at the $i$-th direction. Then $v_1 = v_2$ except at sets with measure $0$.

To say weak derivatives are stable under limits means that if we have a sequence of functions that converges to $u$ and their corresponding weak derivatives converge to $g$, then $g$ is a weak derivative of $u$, as shown in Lemma 3:

Lemma 3. Let $u_n$ be a family of functions in $L^2$ with weak derivatives ($\partial_i u_n$) also in $L^2$ with the following properties:

Then $\partial_i u = g$ (weakly).

This property does not hold for normal derivatives because differentiable functions do not necessarily converge to a differentiable one. An example is:

which is differentiable, but it converges to $u(x) = \abs{x}$ which is not.

Higher-order

So far we’ve only defined the (partial) weak derivative for a single dimension. We can however extend this to multiple dimensions and cardinality. We define a multi-index as:

and the derivative $D^\alpha$ as:

the generalization of weak derivatives is then:

Sobolev Space

Now that we’ve explored weak derivatives, it’s time to make it a proper vector space. The elements of this space are the weakly differentiable functions of order $k$ and in $L^p$. As we introduced in Function Spaces above:

When we say $m$-th derivative here we mean $D^\alpha$ for $\abs{\alpha} = m$. We can define the norm of a function in this space as:

Since norms are dependent on specific vector spaces we can include it to make it clearer:

So in English, the norm of a vector in a Sobolev space is the “length” of the vector formed by all possible partial derivatives adding up to order $k$. For example, if $p = 2$, then it becomes the Euclidean norm.

With this norm, Lemma 4 shows that Sobolev spaces are Banach spaces.

Lemma 4. Sobolev spaces are Banach spaces.

For $W^{k, 2}$ we can define the inner product as:

The norm induced via $\langle u, u \rangle$ is:

the integral is now the $L^2$ norm for $D^\alpha u(x)$ so

which is consistent with the norm defined before. This is enough to show that $W^{k, 2}$ is a Hilbert space. In this context it’s common to use the notation $H^k = W^{k,2}$.

Conclusion

I spent a lot of time (with ChatGPT) trying to get an intuition for weak derivatives and am still not truly satisfied. I do get the idea that we relax the conditions for a derivative by defining it via one of its properties instead.

The same idea is used in vector spaces: Hilbert spaces are those with an inner product. Inner products can induce a norm, but norms can be defined independently of inner produts. Thus, Banach spaces only require norm, not inner product, being thus more general than Hilbert spaces.

The same idea appears in topology e.g. defining things in terms of open sets instead of Euclidean distance. The part I don’t understand is why this property specifically. Why was it chosen over any other property? Maybe once I learn more about its applications in physics I’ll get a better sense.

I’ve read seen Sobolev spaces mentioned in several occasions, especially when reading about physics, but I had no idea what they meant, except that they sound cool. I’m glad to finally understand them a little better.

References

• [1] ChatGPT
• [2] NP-Incompleteness - Functionals

In this post I’ll share my notes on the book Stream Processing with Apache Flink by Fabian Hueske and Vasiliki Kalavri.

This book covers many aspects of the popular open-source Apache Flink, a stream processing engine.

Book Summary

I read the 1st edition of the book. It has 292 pages including the appendix and index, divided into 11 chapters. I skimmed most of the discussions about configuration since I’m mostly interested in Flink’s high-level architecture (I don’t use Flink day-to-day).

Stream Processing. Chapters 1 and 2 provide an introduction to general stream processing concepts. The contents overlap a bit with the book Streaming Systems by Akidau et al.

Architecture. Chapter 3 covers the architecture of Flink at a high level, though the chapters introducing specific features go in more depth on some components.

Features. Chapters 5 to 8 cover the many different features Flink offers. Including the Scala API.

Operations. Chapters 4, 9 and 10 are mostly for operations and configuration. I skipped most of the contents.

Table of Contents

1. Table of Contents
2. Overview
3. Architecture
   1. Components
   2. Data Model
   3. Data Transfer
   4. Watermark
   5. State
   6. Checkpointing
   7. E2E Exactly-Once Semantics
4. API
   1. Stateless Transformations
   2. Keyed Transformations
   3. Windowed Transformations
   4. Multistream Transformations
   5. Join Transformations
   6. Distribution Transformations
   7. Late Event Handling
   8. Operator State
5. Conclusion

Overview

Terminology. A job is the highest level unit an end user interfaces with. So for example when users write a Scala code or a SQL query, it gets mapped to a single job.

A job gets compiled into an execution graph, a DAG of operators. Each operator is in theory executed by multiple tasks (hosts X threads) but in practice operators get fused so the same task ends up processing a chunk of the sub-DAG.

Architecture

Components

JobManager. This is a process that runs in one of the hosts/nodes of the cluster. For fault-tolerance, standby instances of it might run in other hosts. This component is responsible for many things such as compiling a query into a DAG, decides how to split it into tasks, decides where tasks are run, task health monitoring, rebalancing work across tasks.

TaskManager. This is a process running on each host (it’s possible to have more than one task manager per host, but let’s assume it’s 1:1). Each task manager has a thread pool (task slots) that is able to handle work from different jobs, assigned by the job manager.

Tasks also need to send data to one another if they’re processing the DAG of the same job and they don’t live in the same host.

ResourceManager. Like JobManager, it’s logically a single process. This one acts as a broker between JobManager asking for resources and TaskManagers providing them. It knows how many task slots each host has and can return that list of hosts to the JobManager.

It can also pass along requests by the JobManager for more task slots, by then requesting the underlying system (e.g. Kubernetes) to add more workers to the cluster.

Data Model

Operator

Data Transfer

Data can be transferred within the same thread, across threads and across hosts. The intra-thread case is when operators are fused and data is passed between operators. The inter-thread or intra-host is when an operator has to be split. The inter-host case is when tasks need to communicate with multiple other tasks such as in shuffling operations.

Watermark

The watermark is a timestamp that should be interpreted as “no event with timestamp lower than this watermark will appear”. We studied watermarks via the paper Watermarks in Stream Processing in a previous post. Watermark is just a heuristic: we cannot for sure predict whether all events with that timestamp have been seen. However, events that have timestamp lower than the watermark are called late events.

In Flink watermarks are emitted alongside normal events. Operators can transform watermarks in different ways. If an operator has multiple upstream operators, the way it merges watermarks is by taking the minimum between them and re-emitting it.

Other operators such as window/stateful make use of watermark. It’s used to determine when to close windows and emit events.

One downside of emitting watermarks as events is that if one of the tasks is not receiving events it doesn’t know which watermark to emit since it’s event-dependent and watermark progression can get stuck.

State

There are two types of state: key-based (keyed state) or operator-based (operator state). The keyed state as the name implies is like a key-value store. When an operator is key-based, Flink guarantees key-affinity (i.e. the same keys are always processed by the same task) by introducing a shuffle operation beforehand.

The overall way state is implemented is that Flink uses a local key-value store like RocksDB to persist the state and periodically snapshots to an external storage. We’ll cover more about this flow in Snapshots.

Keyed state. The value of the state can be a scalar (ValueState), a list (ListState) or a map (MapState). The reason to expose these as opposed to just working with a strongly typed object is due to the different optimizations they afford.

The scalar is the most general since everything can be serialized to one. But suppose you have a map value and you want to just update one of its keys. If using a scalar, Flink would need to read the entire map, deserialize, update one key, serialize and write back. By using a map state, Flink might decide to store the entries of the map as separate entries in the key-value store.

Operator state. has 3 types: List, Union and Broadcast. These mostly affect the semantics on repartitioning. The list state declares that the entries on the list are largely independent. Once repartitioning happens, Flink might decide to split the list across multiple tasks.

The Union is also a list, but upon restart Flink will union the lists across all tasks and provide each task with this unioned view. The broadcast state is typically sent by an upstream broadcast operator, and read-only for the downstream. On restart each task receives a copy of the broadcast state.

State Evolution. Flink supports some schema evolution if the application uses a backward-compatible serialization such as Avro. It also supports topology evolution as long as operator names are stable (so that their state is preserved). Applications can explicitly label an operator, for example:

stream
  .map(...)
  .uid("user-enrichment")
  .name("User Enrichment Step")

Checkpointing

The JobManager decides to occasionally checkpoint the state of the application. It sends a message to all TaskManagers that then inject a checkpoint barrier, a special message like the watermark.

Upon receiving the barrier, each operator takes a snapshot of its state (the ones discussed above, or offsets for source operators). They have been continuously writing to a local RocksDB and now they need to update that state to an external storage such as S3. This process is asynchronous.

Once the TaskManager determines all operators have uploaded their state, they inform the JobManager. Once the JobManager determines that all tasks have uploaded their state, it can mark the snapshot as completed.

State exactly-once semantics. This 2-phase checkpointing allows for exactly-once semantics of the state (caveat: it assumes operations do not have side effects outside the state). Suppose it checkpointed successfully at time T, then it processed some data and the process crashed. Now it needs to rewind to the old checkpoint, meaning each operator will restore its state from the external storage, then it reprocesses the data.

From the state perspective, no data has been processed more than once. Note that by default Flink doesn’t coordinate flushes to the sink with checkpoints: it constantly flushes data downstream, so the exactly-once semantics is only for state. It’s possible to achieve end-to-end exactly once semantics, but the sink must support it.

E2E Exactly-Once Semantics

Note: for this section, it’s worth being familiar with how Kafka works.

One common way to achieve end-to-end exactly-once semantics in Flink is by using Kafka as a source and sink in EO mode. An important aspect is that Flink does not rely on Kafka to store the topic offset but it stores it as part of its state.

The sink must have exactly once mode enabled and downstream consumers must only read committed logs (see Exactly-Once Semantics in Kafka: The Definitive Guide).

The flow is: Suppose Flink just performed a checkpoint of its state. It then starts a new transaction on the Kafka sink. As it reads data from Kafka source, and processing it, it writes data to the transaction. Once it successfully checkpoints again, it commits the Kafka transaction.

If the process crashes any time before the transaction is closed, it will simply abort the transaction and replay the data from the previously stored offset. One corner case is if Flink crashes after completing its checkpoint but before committing the sink transaction.

To support this case, the transaction handle is stored as part of the checkpointed state. So upon recovery, the first thing it needs to check is whether there is a pending transaction, in which case it would commit it. Then it doesn’t need to reset to the previous checkpoint.

API

Let’s start with a simple example reading data from a socket, transforming it and printing to stdout:

// Read data
val input: DataStream[String] = env.socketTextStream("localhost", 9999)

// Deserialize
val sensorData: DataStream[SensorReading] = input
  .map { line =>
    val parts = line.split(",")
    SensorReading(
      parts(0).trim, // id
      parts(1).trim.toLong, // timestamp
      parts(2).trim.toDouble // temperature
    )
  }

// Transform
val avgTemp: DataStream[SensorReading] = sensorData
  .map( r => {
    val celsius = (r.temperature - 32) * (5.0 / 9.0)
    SensorReading(r.id, r.timestamp, celsius)
  })
  .keyBy(_.id)
  .timeWindow(Time.seconds(5))
  .apply(new TemperatureAverager)

// Print
avgTemp.print()

This example covers several components of a pipeline: reading a serialized message from source, deserializing, transforming and writing somewhere (in this case stdout).

Next we consider the API in more detail.

Stateless Transformations

Map. Is a method of a datastream that takes a function object which itself transforms one message into another message. Example

stream.map( r => r.id )

Filter. Same idea as the map(). Takes one message, returns true/false. If false, the message is filtered out. Example:

stream.filter( r => r.temperature >= 25 )

FlatMap. A more general version of map takes one row but can return a different number of rows. Can be used to implement both map() and filter(). Example:

stream.flatMap { r =>
  if (r.temperature >= 25) List(r.id)
  else List.empty
}

Custom Functions. Instead of passing callbacks to the mentioned APIs, it’s also possible and sometimes necessary to implement interfaces and pass function objects instead. For example, for the .filter() method we can implement:

class FlinkFilter extends FilterFunction[String] {
  override def filter(value: String): Boolean = {
    value.contains("flink")
  }
}
...
var flinkTweets = tweets.filter(new FlinkFilter)

There are corresponding interfaces MapFunction and FlatMapFunction functions for the other 2 APIs. You can also pass parameters to the function upon construction:

var flinkTweets = tweets.filter(new KeywordFilter("flink"))

But there’s “compile” time parameters that will be resolved before the DAG is constructed. For runtime initialization, one can use the Rich- versions, e.g. RichFilterFunction which have the open() and close() methods.

Keyed Transformations

KeyBy. This method on a normal DataStream transforms it into a KeyedStream which is logically partitioned by a key function provided, e.g.

stream.keyBy(r => r.id)

Rolling Aggregations. A rolling aggregation doesn’t require a window of time, the values are accumulated forever, but events are emitted for every row. For example:

stream
  .keyBy(r => r.key)
  .sum(r => r.value)

For each event r, it will add r.value to the corresponding sum for r.key and emit the result.

Reduce. This API allows providing custom aggregation functions. The major restriction is that the accumulator type must be the same as the record type. For example:

stream
  .keyBy(r => r.key)
  .reduce((acc, r) => Record(acc.key, acc.value + r.value))

Note that this API does not have an “initializer” step. The variable acc is set with the first event it sees.

KeyedProcessFunction. The most general functions on KeyedStream are those implementing KeyedProcessFunction. The book provides a complete example, but for brevity, we mention the two methods that must be implemented: processElement() and onTimer().

In processElement() we receive one message and decide what to do with it. This method has access to the (keyed) state, so it can update the state based on this message. This method cannot output messages downstream. Instead it schedules a timer that when fired calls the onTimer() method.

The way to use a custom KeyedProcessFunction is passing it to the process() method:

val alerts =
  readings
    .keyBy(_.id)
    .process(new MyKeyedProcessFunction)

Stateful Map Functions. It’s possible to use keyed state in the generic flat map function, in particular via the RichFlatMapFunction. In this case the state is managed by the function itself and has no timer semantics - the only “hook” point is when flatMap().

class TemperaturAlertFunction()
  extends RichFlatMapFunction[SensorReading, (String, Double, Double)] {

  ...
  override def open(parameters: Configuration): Unit = {
    var lastTempDesc = new ValueStateDescriptor[Double]("lastTemp", classOf[Double])
    lastTempState = getRuntimeContext.getState[Double](lastTempDesc)
  }

  ...
  override def flatMap(reading: SensorReading, out: Collector[(String, Double, Double)]): Unit = {
    ...
    val lastTemp = lastTempState.value()
    ...
  }
}

The state used is of type ValueState. As discussed in Architecture > State > Keyed State, we can also have ListState or MapState. The API will be similar, for example ValueStateDescriptor becomes ListStateDescriptor and getRuntimeContext.getState() becomes getRuntimeContext.getListState().

Windowed Transformations

A special case of keyed transformations is the windowed transformations, on top of WindowedStream. This type of stream is returned once we call methods such as window(). The ProcessWindowFunction interface has the method process() which among other things takes the key and a list of messages within a window.

It’s a layer of abstraction above the KeyedProcessFunction. In this case we don’t have to deal with timers explicitly. The method window() takes a function that describes the window, for example, for a sliding window of 1h and step 15min:

val alerts = readings
  .keyBy(_.id)
  .window(SlidingEventTimeWindows.of(Time.hours(1), Time.minutes(15)))
  .process(new MyProcessWindowFunction)

This can load a lot of data in memory. An alternative is to use either ReduceFunction or AggregateFunction. The former works more like the reduce() for keyed streams, where it receives the accumulation for a particular window and a new element to be added to the window. The latter is a bit more general, you can have different types between the accumulator and the event, but it’s more complex to implement.

It’s also possible to customize what is passed to the .window() API by implementing a WindowAssigner. This determines to which window an event is assigned to. One of the methods in WindowAssigner is getDefaultTrigger() which returns a Trigger.

The trigger tells us when to emit events from the window, and it’s possible to provide a custom trigger that overrides the default for the window, e.g.

val alerts = readings
  .keyBy(_.id)
  .window(new MyWindow)
  .trigger(new MyTrigger)
  .process(new MyProcessWindowFunction)

Multistream Transformations

Union. Merges data of two or more streams of the same type. Events are processed in FIFO order and all events are emitted as they arrive. Example:

var unioned = stream1
  .union(stream2, stream3)

Connect. It combines two streams into a special type of stream called ConnectStreams[T1, T2]. The streams are still technically separated, and are handled by different functions, but the key is that they’re processed by the same operator, so they can share state. Example:

var connected = stream1.connect(stream2)

connected.map1(f1).map2(f2)

So if we decide to store events from stream1 in a state inside map1(), then stream2 would have access to it. Typically we want to shard events by key:

var connected = stream1
  .keyBy(keyFunc1)
  .connect(stream2.keyBy(keyFunc2))

To make sure they’re processed by the same task since state is task-scoped.

Join Transformations

Interval Join. This joins matching events from different streams that are within a period of time of each other:

var connected = stream1.keyBy(_.id)
  .intervalJoin(stream2.keyBy(_.id2))
  .between(Time.seconds(-5), Time.seconds(10))
  .process{ (r, a) => (r, a) }

Where .process() takes a function (lambda or a function of type ProcessJoinFunction) which takes a pair of matching events. The detail is that this join keeps a buffer of the left and right events that are within range. When a new event $e$ arrives, say from the first stream, it emits all pairs containing $e$ and events from the second stream in the buffer.

Window Join. This combines two streams into a (custom) window and performs a join between them. For example:

var connected = stream1.keyBy(_.id)
  .join(stream2.keyBy(_.id2))
  .where(_.id)
  .equalTo(_.id2)
  .window(TumblingEventTimeWindows.of(Time.seconds(10)))
  .apply{ (r, a) => (r, a) }

Where .apply() takes a function (lambda or a function of type JoinFunction) which takes the cross product of the list of elements from the first stream with those of the second within a window of time.

Distribution Transformations

By default, when using keyBy(), Flink will distribute data to the right tasks based on the provided key (expression). There are ways to customize that using APIs such as shuffle(), rebalance() and rescale().

Late Event Handling

Due to watermark, some events will be considered late. There are many options on how to handle them: the simplest is discarding them, but an alternative is sending them to a different sink (dead letter queue).

This can be done via the .sideOutputLateData():

stream
  .keyBy(_.id)
  .timeWindow(Time.seconds(10))
  .sideOutputLateData(new LateEventSink)
  .process(new MyProcessFunc)

When customizing ProcessFunction we can also handle late events since this function has access to the watermark.

Operator State

For non-keyed state, we discussed a few options in Architecture > State > Operator State: List, Union and Broadcast. A flat map function can use them by using the trait ListCheckpointed, which requires implementing the restoreState and snapshotState. For example:

class MyStatefulFunction()
  extends RichFlatMapFunction[SensorReading, (Long)]
  with ListCheckpointed[java.lang.Long] {

  ...
  override def restoreState(state: List[Long]) = { ... }
  ...
  override def snapshotState(chkpntId: Long, ts: Long): List[Long] = { ... }
}

To use other types of state such as union, we must use the more general trait, CheckpointedFunction:

class MyStatefulFunction()
  extends RichFlatMapFunction[SensorReading, (Long)]
  with CheckpointedFunction {

  ...
  override def initializeState(ctx: FunctionInitializationContext): Unit = {
    val desc = new ListStateDescriptor[String]("rules", classOf[String])
    unionState =
      ctx.getOperatorStateStore.getUnionListState(desc)
  }
  ...
  override def snapshotState(ctx: FunctionSnapshotContext): Unit = { ... }
}

Conclusion

I’ve been working with stream processing for about 4 years and never took the time to learn about the most popular open source stream processing system.

One thing that I found surprising is how expressive Flink is, and how many different levels of abstractions are supported in the APIs. This also means there are many ways to do the same thing, which I tend to not be a big fan of.

I started reading the book The Theoretical Minimum by Leonard Susskind and George Hrabovsky. A lot of the math from the early chapters looked familiar, but in Chapter 6: The Principle of Least Action, they describe and derive the Euler-Lagrange equation, which I don’t recall seeing before.

I wanted to explore these equations and their derivation, but from a more mathematical point of view. This led me to a short rabbit hole around functionals and Sobolev spaces and since I like learning things from first principles, I decided to cover functionals first.

The thumbnail features the Hungarian mathematician Frigyes Riesz. He was first featured in the post Subharmonic Functions and is back here because he’s considered one of the founders of functional analysis and we’ll cover one theorem named after him, the Riesz Representation Theorem.

Functionals

In the branch of math called functional analysis the core object is the functional. Without qualifiers it’s implicitly assumed that a functional is a linear one. In this post however, since we want to cover functionals in general, we’ll qualify linear functions explicitly and assume the general case when saying functionals.

A functional is essentially a function $f$ where the domain is a vector space $H$ and the image is a field, either the reals or the complex numbers, generically denoted by $\mathbb{F}$:

More intuitively, a functional is a function that takes a function as input and returns a scalar. Much like functional programming which operates over functions as objects.

One may ask: what do functions have to do with vector spaces? In a vector space the vector is a concept more broad than, say, a tuple of scalars like $\mathbb{R}^3$. It just means it’s some object that satisfies a set of axioms (e.g. addition, scalar multiplication). An example of a function vector space is the set of continuous functions.

If this function (or map) satisfies additivity and scalar multiplication, then it’s called a linear functional. In other words if $x, y$ are members of a vector space $V$, and $\lambda \in \mathbb{F}$ and $f$ a linear map $f: V \rightarrow \mathbb{F}$:

We’ll now cover some related concepts and properties of functionals.

Properties

Norm

The norm or operator norm of a functional $f$ and denoted by $\norm{f}$ is defined as:

That is, it’s the supremum of the value of $f$ but normalized by its input size. Note that it doesn’t make sense to talk about $\abs{f}$, even though its image is a scalar, since it requires a specific input for it to spit out a scalar.

Continuity

For continuity to make sense for functionals, the domain must be a topological space, i.e. it must have the notion of open sets, because continuity depends on these.

In a more specific case, if we assume the domain is a normed vector space, i.e. it has the notion of distance between its elements, then we can use the $\epsilon-\delta$ definition of continuity for the functional, that is, a functional is continuous at $x_0$ if for every $\epsilon \gt 0$, there exists $\delta \gt 0$:

For a linear functional $f$ in particular, there’s an alternative characterization: if there’s an upper bound on how much bigger $f$ is compared to its input, then it’s continuous:

Lemma 1. The linear functional $f: H \rightarrow \mathbb{F}$ is continuous if and only if

for all $x \in H$ and some constant $C$.

Differentiation

Let $X, Y$ be vector spaces equipped with a norm and $f : X \rightarrow Y$. We say that $f$ is Fréchet differentiable if there exists a linear map $A: X \rightarrow Y$ (Fréchet derivative) such that:

For $h \in X$. Note that $f$ is not necessarily a functional, only if $Y = \mathbb{R}$ or $Y = \mathbb{C}$, and even if it is, it’s not necessarily linear. However if $f$ is a functional, then the Fréchet derivative is a linear function because it’s a linear map from a vector space to a field.

This is a general definition of the differential we see in real analysis. If we take $X = Y = \mathbb{R}$, then we can simplify $(1)$ to:

By having $A(h) = f’(x)h$. If we add $f’(x)h/h$ to both sides of the equation, we get the more familiar:

One might ask why we use this other form. The expression $f(x + h) - f(x)$ is an element of $Y$ and $h \in X$, so in order for $(3)$ to make sense, we’d need to define multiplication or division between $X$ and $Y$.

The Fréchet derivative also generalizes the complex derivatives, the one that defines holomorphic functions and underpins complex analysis. We can also have the form $(2)$ but the implicit assumption is that in $f’(x)h$ the multiplication operator is the complex one.

Note how the Fréchet derivative goes one abstraction layer above by “wrapping” $f’(x)h$ as some function $A(h)$. This is similar to how topological spaces abstract normed spaces by working with open sets instead of norms (open set is a higher object than norms because they can be defined from norms but not the other way around).

Linear Functionals

Now we focus on properties that are only applicable if the functional is linear.

Kernel

Let $X, Y$ be vector spaces and $f$ a linear map between them. The kernel is the subspace of $X$ defined as:

In other words, all elements in the domain that map to $0$ in the image. Note that $0$ here is not necessarily the scalar number $0$, but the $0$ element in the vector space $Y$.

We can verify that the kernel is indeed a subspace of $X$. It contains the $0$-th element because $f$ is linear and thus $f(0) = 0$. If $x, y \in \ker f$, then $x + y \in \ker f$ again because $f$ is linear and $f(x + y) = f(x) + f(y) = 0 + 0$. If $x \in \ker f$ and $\lambda$ is a scalar, then $\lambda x \in \ker f$ because $f(\lambda x) = \lambda f(x) = 0$.

If $f$ is continuous, then $\ker f$ is closed. We can show this by using one of the topological definitions of continuity: A function $f: X \rightarrow Y$ is continuous if and only if for every $U$ that is a closed set in $Y$, $f^{-1}(U)$ is a closed set in $X$. Since $\curly{0}$ is a closed set in $Y$ and $\ker f$ is the pre-image of $\curly{0}$, $\ker f$ is closed.

So we know the kernel is a subspace of $X$, but it doesn’t necessarily inherit the same properties of the vector space $X$. If the domain is a Hilbert space however, then it’s possible to show that $\ker f$ is also a Hilbert space.

Dual Space

Intuitively things that have linear properties can form a vector space, because a lot of its axioms are about linear combination of vectors. Since linear functionals have linear properties, they can also form a vector space! This vector space is called the dual of the domain vector space of the functionals.

More specifically, the dual space of a vector space $V$ is the set of all linear functionals that have $V$ as domain, denoted with a superscript asterisk:

where $f$ is a linear functional. The intuition here is that $f$ associates a measure to the vectors of $V$. For Hilbert spaces in particular, we have a nice identity between a vector space and its dual. Before we show that, we need the following result:

Theorem 2. (Riesz Representation Theorem) Let $H$ be a Hilbert space and $f: H \rightarrow \mathbb{F}$ a continuous linear functional. Then, there exists a unique vector $y \in H$ such that:

and with $\norm{f} = \norm{y}$

What this theorem is saying is that for any continuous linear functional $f$ over a Hilbert domain, there’s exactly one element in that domain that “encodes” $f$ as a dot product with that element.

We can now claim that Hilbert spaces are isomorphic to their dual, $H \cong H^*$, that is, there exists a bijection between these two sets. In this case the function is defined by:

which is a bijection since for each $f \in H^*$ there’s a unique $y$ for which $f = f_y$. Conversely each $y$ defines a unique function $f_y$. Further since $\norm{f} = \norm{y}$ this is a “length-preserving” bijection, which leads to the more general isometric isomorphism.

This is a special identity for Hilbert spaces. It does not hold for example for the slightly more general Banach space.

Conclusion

I thought this would be my first post on functional analysis, but I had forgotten I wrote about Hilbert spaces before.

This was another topic I relied entirely on ChatGPT and really liked the interactive process. It’s very gratifying to start with a blurry view of it and gradually build a cohesive and more intuitive picture.

One of the most amusing moments was when I asked ChatGPT what happens if we go recursive and build a vector space of functionals, and then it pointed out it’s basically the dual space, which I had already studied. It then “clicked”.

It was also nice to connect with other parts I have studied in the past such as analysis and topology.

In this post I’ll share a summary on the book Complex Analysis by Lars V. Ahlfors, and my journey in studying it.

I’ll start with the journey because I think it’s the more interesting. The second part is basically a link to all the posts I wrote from studying this book.

The Journey

I’ve wanted to study Complex Analysis since I read Jay Cummings’ Real Analysis: A Long-Form Mathematics Textbook. The book is very approachable and I finally understood many bits I had learned from Calculus. I still recall the “wow” moment when I saw the formal definition of integral.

The opportunity came when I was visiting Taichung in Taiwan and visited a bookstore called Mollie Used Books. There were only a few books in English and I was baffled to find a math textbook among them, so I had to get it. The book was only $4 if I recall correctly.

In retrospect, that was one of the biggest “penny wise, pound foolish” mistakes of mine. Complex analysis is a hard subject and Ahlfors is not beginner friendly.

I should have started with Tristan Needham’s Visual Complex Analysis, but I fell prey to the sunken cost fallacy: I ended up buying Needham’s book but stuck with Alfhors as the main guide. I did learn about multi-valued functions from Visual Complex Analysis because I couldn’t grok Alfhors’ explanations.

I had to complement my readings online with Mathematics on StackExchange and scattered lecture notes. But because these notes approach subjects in different angles and order, many times it wasn’t easy to make use of them to understand the book. A notable example was when studying Zeros and Poles. Terence Tao’s notes was much more intuitive than the book, but he relies on a different foundation so I had to study a bunch of things such as Laurent series out of order.

As ChatGPT appeared, I started using it more and more and largely replaced other sources of complement. I was using it inefficiently though: I would still try to follow Alfhor’s proofs but ask ChatGPT about a leap in argument but often times ChatGPT wouldn’t be able to help because it was something very specific with the proof and my edition.

Only more recently I found that I should rely on ChatGPT for the entire proof, which might be completely different from the one in Alfhors. One major benefit of using ChatGPT is that the initial proof it provides is very high level and intuitive but skims details, which I can then dig into.

It took me roughly 2 years and a half to finish this book. I’d estimate I spend 2h on average per week working through it, so about 260 hours.

Summary

Chapter 1: Complex Numbers

Covers the most basic properties of complex numbers.

Posts from this chapter:

• The Cardinality of Complex Numbers
• Complex Numbers and Geometry
• The Gauss-Lucas Theorem

Chapter 2: Complex Functions

Introduces functions of complex numbers. In particular holomorphic functions which are complex differentiable functions. Alfhors calls holomorphic functions analytic, which seems to be an outdated terminology. They have different definitions, but it can be shown that analytic functions are the same as holomorphic functions.

Posts from this chapter:

• Holomorphic Functions

Chapter 3: Analytic Functions as Mappings

This chapter reviews point set topology, defines conformal maps (angle-preserving transformations) and Möbius transformations which he calls linear transformations.

Posts from this chapter:

• Conformal Maps
• Möbius Transformation
• Cross Ratio
• Circles of Apollonius
• Symmetry Points of a Circle
• Bipolar Coordinates and Möbius Transformations

Chapter 4: Complex Integration

As the name suggests, complex integration is defined here. Most of the times we’re interested in integration over a 1d-curve embedded in the 2d-complex plane (line or contour integrals). It seems common to denote contour integrals as $\oint$ but I used Alfhors notation of the simple integral symbol $\int$ with the contour being implied by the subscript path, e.g. $\int_\gamma$.

Here’s where Cauchy’s name start popping up everywhere. This seems the longest and most information dense chapter of the book.

Posts from this chapter:

• Complex Integration
• Path-Independent Line Integrals
• Cauchy Integral Theorem
• The Winding Number
• Cauchy’s Integral Formula
• Removable Singularities
• Zeros and Poles
• The Open Mapping Theorem
• The Maximum Principle
• The General Form of Cauchy’s Theorem
• The Residue Theorem
• Harmonic Functions

Chapter 5: Series and Product Development

This covers infinite series and products. Here’s where a bunch of surprising connections star to appear, especially with Weierstrass Factorization Theorem and The Riemann Zeta Function. This was my favorite chapter.

Posts from this chapter:

• Runge’s Theorem
• Mittag-Leffler’s Theorem
• Holomorphic Functions are Analytic
• Weierstrass Factorization Theorem
• The Gamma Function
• Hadamard Factorization Theorem
• The Riemann Zeta Function

Chapter 6: Conformal Mapping. Dirichlet’s Problem

Here the book discusses the Dirichlet’s Problem: how to evaluate a holomorphic function if we only know its values at the boundary of a closed curve. The conformal mapping part helps reducing the problem to simpler curves but I found it a bit tedius. Perron’s method to solve a special case of Dirichlet’s Problem was very interesting.

This was my least favorite chapter. Posts from this chapter:

• Subharmonic Functions
• The Perron Method

Chapter 7: Elliptic Functions

This chapter felt a bit more digestable than the previous one, back to some elementary ideas such as periodic functions. I got a glimpse of the connection with number theory and it tempted me to study number theory in more depth.

Posts from this chapter:

• Elliptic Functions
• The Weierstrass ℘ Function

Chapter 8: Global Analytic Functions

I skipped this chapter entirely. The gist seems to be that for all the results we studied it’s assumed the functions are (locally) single-valued, so we often have to restrict the domain to specific branches which makes it imposible to have a single function that works for the whole domain.

This chapter discusses things like germs and sheaves that help us lift the single-valued constraint and allow defining a single function for the entire domain.

I skipped because this feels like an entire new beast with new terminology and ideas, and because I’m tired. I believe this is very important in fields like algebraic geometry, so if I ever muster the courage to study it, I’m hoping I’ll learn about this properly.

Final Thoughts

Despite the time sunk and the frustration of feeling dumb for not being able to understand the book many times, I am very appreciative of what I have learned. I think this is hard stuff and I’m proud at myself for sticking with it for so long.

It feels good being able to experience this knowledge that only few people (which I presume most are being major in math) had. It’s like going on a very long hike and getting very far, seeing many beautiful things along the way, while most people only explore the beginning and miss most of them.

Would any of this be useful? I have no idea. But I do hope I now have a stronger foundation to understand more practical fields such as AI and p hysics.

In our post on Elliptic functions [2] we started with the simply periodic functions such as $\sin z$. We noted that $e^{2\pi i z} / w$ is the simplest of the periodic functions and that every single simply periodic function $f(z)$ of period $w$ can be written as a function of it: $f(z) = g(e^{2\pi i z} / w)$.

Then we introduced doubly periodic functions, also known as elliptic functions. One may ask if there is, analogously, the simplest elliptic function and whether it’s possible to write all elliptic functions as a function of it. The answer is yes! And this function is known as the Weierstrass ℘ function which we’ll study in this post.

Fun Trivia

The symbol ℘ denoting the Weierstrass elliptic function seems to be the only unicode character that represents a very specific mathematical object.

Even symbols you might associate with specific meanings, such as π, are just Greek letters, so they’re not exclusively used to represent a constant. Even glyphs like ℝ, often associated with the set of real numbers, are just conventions.

Physics has its counterpart, ℏ, denoting Planck’s constant.

The Simplest Function

In our post on elliptic functions [2], we showed that an elliptic function without poles is a constant and that an elliptic function cannot have a single simple pole. Thus the simplest interesting function is one that has order 2. In this case it either has 2 distinct poles or it has a single pole with order 2, which is also known as a double pole. We’ll show that the Weierstrass $\wp$ function is an elliptic function with a double pole.

Let $f(z)$ be an elliptic function with period module $M$. We defined the Weierstrass $\wp$ function as

We start by showing (Lemma 1) that the series converges, and thus that $\wp$ exists.

Lemma 1. The series in $(1)$ converges uniformly on every compact set $K \subset \mathbb{C} \setminus M$.

Now we need to show that $\wp$ is an elliptic function with a double pole. We note that the only poles of

are when $z = w$, thus the periods in $M$. So in a compact set that avoids the lattice points induced by $M$ we get that $(2)$ is holomorphic. The Weierstrass convergence theorem states that if $f_n(z)$ is holomorphic, and the series $\sum_{n} f_n(z)$ converges uniformly, then this series is also holomorphic.

So in some carefully chosen region $(2)$ is holomorphic and thus the series in $(1)$ is a meromorphic function with a double pole at $z = 0$.

Corollary 2. The function $\wp$ has a double pole at 0.

With that information we just need to show:

Lemma 3. $\wp(z) = \wp(z + \lambda)$ for any $\lambda \in M$

to show $\wp$ is elliptic.

So we’ve defined the Weierstrass function and have shown it’s elliptic with a double pole at 0. Now we consider some other properties.

Properties

Because $\wp$ has only one double pole at 0, we can write its Laurent Series expansion as:

We can then use Lemma 4 to show that $\wp$ is even:

Lemma 4. Let $f(z)$ be a function with the Laurent expansion as in $(3)$. Then $f(z) = f(-z)$.

Another conclusion from the Laurent series is that $\wp$ has residue 0 [4], because by definition it’s the coefficient $c_{-1}$ of the Laurent series.

Corollary 5. The function $\wp$ has residue 0.

We now explore the anti-derivative of the Weierstrass function.

Weierstrass Zeta Function

Recall that the anti-derivative $g$ of a function $f$ is one such that $g’ = f$. For the Weierstrass function, its anti-derivative is known as the Weierstrass Zeta Function and denoted by $\zeta$, but conventionally we work with the negative of the anti-derivative, so $\zeta’(z) = -\wp(z)$.

We can compute it from $(1)$ by integrating over the terms of the series, as shown in Lemma 6.

Lemma 6. The Weierstrass Zeta Function can be written as

The Laurent series of $\zeta$ around 0 has the principal part $1/z$ and the rest, $h(z)$, is a holomorphic function:

this means this function has residue 1 around 0. Lemma 7. generalizes this for all other periods in $M$.

Lemma 7. The function $\zeta$ has residue 1 around $w \in M$.

From Lemma 40 in [2], which claims that the sum of residues of an elliptic function is 0, this implies that $\zeta$ is not a periodic function. It is however, as Lemma 8 shows, a quasi-periodic function. A quasi-periodic function $f$ is one for which $f(z) = f(z + w) + \eta_w$ for some constant $\eta_w \in \mathbb{C}$ dependent on $w$.

Note that a quasi-periodic function is a weaker version of periodic functions, since the latter are quasi-periodic functions with $\eta_w = 0$.

Lemma 8. The function $\zeta$ is quasi-periodic.

There’s a nice identity between the base of the period module and the corresponding quasi-period constants for the zeta function, known as Legendre’s Relation. Lemma 9 has more details:

Lemma 9. Let $(w_1, w_2)$ be a base for the period module $M$. Let $\eta_1, \eta_2$ be the constants in $\zeta(z + w_1) = \zeta(z) + \eta_1$ and $\zeta(z + w_2) = \zeta(z) + \eta_2$. Then:

Lemma 10. The function $\zeta(z)$ is odd.

Weierstrass Sigma Function

Another important function related to $\wp$ is Weierstrass sigma function, denoted by $\sigma$ and defined as a function of which $\zeta(z)$ is the log derivative of. More precisely, we have that:

This function has a specific product form, as shown in Lemma 11.

Lemma 11. The Weierstrass sigma function can be expressed as

Quasi-periodicity doesn’t have to be additive, it can also be multiplicative. In the sigma function case it’s possible to show

The derivative of $\wp(z)$

The derivative $\wp’(z)$ can be expressed as:

Lemma 12.

From this expression we can see the poles of $\wp’(z)$ are the same as $\wp(z)$, i.e. the periods in $M$. Also, by plugging $z$ and $-z$ in this formula, we can conclude that

Corollary 13. The function $\wp’(z)$ is odd.

By Lemma 3 in [2] which says that elliptic functions are closed under derivation, we conclude that:

Corollary 14. The function $\wp’(z)$ is elliptic with the same period module $M$.

We now characterize the zeros of $\wp’(z)$, via Lemma 15:

Lemma 15. Let $a \not \in M$. Then $\wp’(a) = 0$ if and only if $2a \in M$.

Elliptic Functions

We’re now ready to show that every elliptical function can be written from $\wp(z)$ and $\wp’(z)$. We handle even functions and odd functions separately, which suffices since any function can be decomposed into even and odd functions [6].

Lemma 16. Let $f(z)$ be an even elliptic function. Then there exists a rational function $R$, such that:

Recall that a rational function is of the form $R(z) = P(z)/Q(z)$ where $P$ and $Q$ are polynomials of $z$ and $Q(z) \ne 0$. Now for odd functions:

Lemma 17. Let $f(z)$ be an odd function. Then there exists a rational function $R(z)$ such that $f(z) = \wp’(z) R(\wp(z))$.

In [6] we showed that every function $f(z)$ can be decomposed into odd and even components:

With

Since elliptic functions are closed under arithmetic operations, we conclude that $f_e(z)$ and $f_o(z)$ are elliptic and can use Lemma 16 to show that $f_o(z)$ can be written as $R_1(\wp(z))$ and Lemma 17 that $f_e(z)$ can be written as $\wp’(z) R_2(\wp(z))$, so $f(z) = R_1(\wp(z)) + \wp’(z) R_2(\wp(z))$ leading to the following:

Corollary 18. Every elliptic function $f(z)$ can be written as a rational function of $\wp(z)$.

Conclusion

In this post we introduced the Weierstrass functions which include the $\wp$, the zeta and the sigma functions. We ended up not using the latter two, but I included them because they’re discussed in Ahlfors.

The proof that every elliptic function can be expressed from $\wp(z)$ was extremely difficult to understand. It has many subtle details and every time I re-read the proof I realized I had glossed over something.

This proof is also not provided in Ahlfors (in any obvious way at least) and I relied entirely on ChatGPT to understand it! Like with learning history, I’m finding using ChatGPT a lot more effective at learning math: I can ask for it to explain me things in different angles and dig into different parts.

Part of me is reluctant on this approach because I really like the idea of reading books, but going forward I might need to rethink my approach. Perhaps I’ll use textbooks to get a general idea of the field but rely more on ChatGPT to really understand things.

This post also marks the end of my journey with the textbook Complex Analysis by Lars V. Ahlfors. I’ll write another post summarizing the book and look back on this journey.

Appendix

Lemma 19. Let $P(z)$ be a polynomial:

then near 0, $1/P(z)$ is of the form:

where $c’_1 = 1/c_1$.

Lemma 20. Let $P(z)$ be a polynomial:

then near 0, $1/P(z)$ is of the form:

where $c’_1 = 1/c_1$.

References

• [1] Complex Analysis - Lars V. Ahlfors
• [2] NP-Incompleteness: Elliptic Functions
• [3] NP-Incompleteness: Zeros and Poles
• [4] NP-Incompleteness: The Residue Theorem
• [5] NP-Incompleteness: Removable Singularities
• [6] NP-Incompleteness: Hermitian Functions

In this post I’d like to share my thoughts on coding with AI and how it has affected me. Everyone is talking about this and I don’t have anything new to say, but want to centralize points I’ve heard/read so far and document this point in time. It might be fun to come back to this after a while.

A New Era

I kept hearing leadership saying AI would fundamentally change how we work and I was initially dismissing it as hyperbolic but then I started using Claude skills last December. I started getting value from it even before the introduction of Opus 4.5: I created a skill to carry out a refactoring that would have taken me days, and it finished the work in a couple of hours with little interaction needed from me.

Now I’m at a point where I don’t type code in the editor anymore, and do everything through Claude code or the terminal. I plan to write more details on how I use AI for my day-to-day work in a separate post.

Loss of Craft

According to Daniel Pink, the three pillars of intrinsic motivations are: autonomy, purpose and mastery. Delegating coding to AI has eliminated a lot of the mastery aspect of writing code. I’ve seen a lot of people at work sad / lost with this change.

I thought I’d be on the same boat because I love coding! I participated in ICPC competitions during college (I spent many school breaks happily practicing), I can do Leetcode for fun and still solve programming puzzles like Advent of Code. I was surprised at myself when I was excited about writing prompts all day.

Reflecting on why, I realize that most coding in a day-to-day job is not that interesting. It revolves around writing boilerplate, fixing compilation/type errors, adhering to convention and working with human-induced complexity and abstractions. I don’t enjoy any of that. I still plan to keep solving programming puzzles for fun.

Skill Atrophy

Some scientific studies show that people who use GPS have worsened spatial navigation abilities. While this is not permanent, if you get lost during a hike, you won’t be able to rely on these skills.

I think a similar pattern will happen in the future for programming. Most people will have trouble writing code by hand and once we delegate code reviews to AI we’ll have trouble reading it too. This will be even worse for people who are joining the workforce now, since they will not even have long-term memory to fall back on.

So in the event of an emergency blackout where we can’t rely on AI, very few people would be able to help, but maybe that’s ok.

Overwork

I saw this blog post by Simon Willison that resonated with me, claiming that people work more with AI. It seems like a paradox given that AI is supposed to automate a lot of our work.

To me, the major reason for overworking is that now I can get work done with smaller chunks of time. Before, if I had a gap of 30 minutes free I’d browse my phone or do something else. Now I have time to write a prompt and have Claude code fix a small issue. @darkzuckerberg on Threads uses the term casual productivity for this.

This also increases the perceived opportunity cost: I feel like I’m constantly thinking of ways I could be leveraging AI to advance some project. This sentiment is also shared by Philip Su:

In honesty, I don’t even use the bathroom these days before prompting several AIs with work while I’m gone 120 seconds.

This also means that my attention span, which is already not great (see Context Switching), is going to get worse.

Career Expectations

The part that I worry about the most is whether this new paradigm will benefit me more or less than the average.

I don’t enjoy writing documents and spending days in meetings to get alignment. While I don’t feel like I need to write the code myself, I do like having more direct contribution to projects, which so far translated into doing the work myself or mentoring more junior engineers / interns where my level of contribution is at the task level. Currently AI seems to be providing most value at this junior level and I’m able to leverage it.

That suggests junior people will be at a disadvantage because any engineer can use AI to do the same work, but junior people often lack the knowledge and experience to know what to build and verify the result. On the other hand, junior engineers are more open to adopting new technologies and are less expensive for the company, so who knows.

The senior engineer who typically delegates more concrete tasks to junior engineers can now do the same to AI agents and get the work done faster. However, the technical lead can also do this and with AI being able to handle more complex workflows, it will be possible to operate at a higher level of abstraction such as writing architectural documents.

My prediction is that the more AI advances, the more it will benefit higher level engineers. I expect it will widen the impact gap between the different levels and I wonder if this will reflect in compensation. This might also expose high-level engineers that are only good at people skills but not good at engineering, because now they might be expected to produce concrete output, even if prototypes.

We’ll see. I’m part anxious, part excited about how things will change in the next year or so.

Mind the Gap

Speaking of widening the impact gap, I expect that – at least while things are being figured out – some people will be vastly more productive with AI than others. Without AI, the ratio of the fastest typer vs. the slowest one is probably not much more than 2x. It’s said that the mythical 10x engineer is often more efficient because they choose to solve a problem in a different way or realize the problem does not need to be solved. However, it’s really hard to prove this efficiency objectively.

With AI, someone skilled at prompting might be able to orchestrate a large project without intervention. They might get it churning during sleep hours or weekends. And I think this could lead to a 100x difference between the most productive and least productive person, at least until tools mature.

Now that prototyping is cheap, people can also be compared more objectively. They could have team A and B compete to work on a major project in parallel, with different approaches. Someone from a different team might vibe code your project during a weekend because they’re most skilled at prompting. This is pretty anxiety-inducing and also plays into overwork (see Overwork above).

Context Switching

Over my career a consistent piece of feedback I got is that I work on too many projects in parallel. I often need a lot of effort to correct course and focus on fewer things, but I’m just excited about too many things and invariably end up relapsing.

I found so far that with AI I can more effectively operate in this mode. I have a main project which I prioritize but when Claude is “Shimmying…” I can context switch to a side quest. I read some people struggle with this but since I’ve been doing this since forever, it feels natural to me.

Relatedly, Boris Cherny, the creator of Claude code, recently did a Q&A and mentioned that we’re in the Golden age for ADHD. I never got diagnosed with ADHD, but I check a lot of the symptoms.

Jevons Paradox

One interesting aspect of individuals becoming 2x, 10x more efficient is what companies will end up doing: reducing work force proportionally or increasing the number and scope of projects.

So far it seems like most of the productivity is in coding, but code review and operational oversight still seem to require human intervention. Also, a lot more changes going in means the surface area that needs to be supported increases.

I think it’s a matter of time before most human oversight will be taken out of the loop and this will mean it will disrupt the software engineering profession. Whether this will cascade to other professions remains to be seen.

Coding Style Changes

It’s said that code is written once and read many times, so we need to optimize for readability. I think many people do the right thing for stuff like naming variables, but I found that when it comes to abstraction we often optimize for writing. The DRY (Don’t Repeat Yourself) principle is often used as justification for excessive abstraction which makes the code short and dense but hard to follow.

AI has no trouble generating tons of code so it can be as verbose as we want. It’s also good at being more consistent and thorough than humans so the argument to avoid duplicate code because it needs to be updated multiple times might not be as strong now. Multiple layers of useless abstraction might also require AI to bring in more context and degrade its performance.

On a similar vein, I’m changing my mind about comments. I often preferred sparse comments because it’s hard to keep them consistent with the code but AI is pretty good at keeping them up to date so I feel like it’s okay to have them now. One thing that AI is still bad at is explaining the “why”. It’s very good at explaining the “what” but it doesn’t explain why it decided to implement things this way until you actually ask it. It probably copied this behavior from training data.

I learned a lot about BPF tools after reading the book BPF Performance Tools by Brendan Gregg. The book focused mostly on performance analysis at a lower level, either through kernel functions or libraries such as libc.

I wanted to leverage BPF at the application layer, in particular in C++. The book covers C++ very briefly in Chapter 12 and Chapter 13 provides an example of analyzing a MySQL database but still, most of the examples assume the implementation being in C.

In this post want to investigate how to inspect C++ applications.

Dynamic Probes

Let’s start with dynamic probes. As we explained in [1], these are probes we can attach to without having to recompile the code. Since we’re only interested in user space, we’ll use uprobes.

Functions

Let’s start with a simple example: free functions. In our example, we run an infinite loop simulating a webserver.

// main.cpp

long long fibonacci(int n) {
  return n <= 1 ? n : fibonacci(n - 1) + fibonacci(n - 2);
}

int main() {
  while (true) {
    std::cout << fibonacci(40) << std::endl;
  }

  return 0;
}

We can run this binary say via

g++ main.cpp -o fibo -O3

and then use this bpftrace script that counts how many times fibonacci() was called during a 2-second period:

// fib.bt

uprobe:./fibo:_Z9fibonaccii { @calls = count(); }

interval:s:2 { exit(); }

We can run it via:

$ sudo bpftrace /tmp/fib.bt

Attached 3 probes
@calls: 9723017

Name mangling. The first hurdle is that C++ mangles the name of functions, even free ones, so we need to provide that instead, _Z9fibonaccii. The easiest way to find the mangled symbols from a binary is by running this bpftrace command:

$ sudo bpftrace -l 'uprobe:./fibo:*fib*'
uprobe:./fibo:_Z9fibonaccii

We can also get a distribution of values passed to fibonacci by changing fib.bt:

uprobe:./fibo:_Z9fibonaccii {
  @h = hist(arg0);
}

interval:s:2 { exit(); }

And then:

$ sudo bpftrace /tmp/fib.bt

@h:
[0]              1720997 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@                    |
[1]              2784630 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2, 4)           2784630 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)           1469908 |@@@@@@@@@@@@@@@@@@@@@@@                         |
[8, 16)           245746 |@                                               |
[16, 32)            5339 |                                                |
[32, 64)               3 |                                                |

Note that because fibonacci is recursive, we don’t need to worry about it being inlined by the compiler optimizer. Otherwise we need to modify the code to add __attribute__((noinline)) if we want to turn off inlining for this function.

Methods

Class methods look like regular functions after compilation and the concept of method visibility does not exist in runtime, so we can also inspect private methods. We can modify our example to implement such a case:

// main.cpp

struct Fibo {
  // Expensive recursive Fibonacci computation
  long long run() {
    return run(40);
  }

 private:
  long long run(int n) {
    return n <= 1 ? n : run(n - 1) + run(n - 2);
  }
};

int main() {
  Fibo f;
  while (true) {
    std::cout << f.run() << std::endl;
  }

  return 0;
}

If we compile it to fibo_class, we can inspect the symbols by looking for the class name:

$ sudo bpftrace -l 'uprobe:./fibo_class:*Fibo*'
uprobe:./fibo_class:_ZN4Fibo3runEi
uprobe:./fibo_class:_ZN4Fibo3runEv

Since we have overloaded signatures, it shows both. We can use c++filt to identify the correct overload, the one taking int as argument:

$ c++filt _ZN4Fibo3runEi
Fibo::run(int)

So we can write a similar script to count the distribution of arguments. In C++, when methods are compiled, the this object that is implicit in code is made explicit as the first argument, so we have to account for that and probe the second argument arg1 instead.

uprobe:./fibo_class:_ZN4Fibo3runEi { @h = hist(arg1); }

interval:s:2 { exit(); }

STL Strings

bpftrace can handle C-style char* but not std::string. For that, we need to make assumptions about which STL implementation is used, operating systems and compilation mode, making this unportable. Suppose we have:

void my_print(const std::string& s) {
  std::cout << s << std::endl;
}
int main() {
  std::string s1 = "hello";
  std::string s2 = "world";

  int i = 0;
  while (++i) {
    i % 3 == 0 ? my_print(s1) : my_print(s2);
    sleep(1);
  }

  return 0;
}

And we want to probe s in my_print. Suppose we compile this to strhist. I’m running this on Linux on a x86_64 architecture and can verify my program links to stdc++

ldd strhist
...
libstdc++.so.6 => /lib64/libstdc++.so.6 (...)
...

In this case it’s relatively safe to assume that if std::string is at the address addr, then addr + 0 is a pointer to the data and addr + 8 contains the length of the string. This works even with SSO (small string optimization) in which the raw data is not stored in the heap, but within the std::string structure itself, in the buffer zone. In that case addr + 0 will not point to the heap, but to addr + 16, but for our purposes it doesn’t matter.

We can write a bpftrace script as such:

// strhist.bt
uprobe:./fibo:_Z8my_printRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
{
  $s  = arg0;
  $p  = *(uint64*)($s + 0);    // char* - raw data
  $n  = *(uint64*)($s + 8);    // size_t - strlen

  // print as raw bytes (handles embedded NULs)
  printf("len=%lu str=%r\n", $n, buf($p, $n));
}

We’ll see it shows hello and world correctly. We can get a frequency count of each:

// strhist.bt
uprobe:./fibo:_Z8my_printRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
{
  $s  = arg0;
  $p  = *(uint64*)($s + 0);    // char* - raw data
  $n  = *(uint64*)($s + 8);    // size_t - strlen

  @cnt[str($p)] = count();
}

The major drawback of str($p) is that it’s truncated, typically 64 bytes, so if the strings share the same long prefix, it will not distinguish between then. Also, if these strings have \0, then it will use as delimiter instead of honoring $n. It’s possible to create a custom hashing function with length larger than 64 but it must be a constant.

User-Defined Classes

If the argument is a more complex class, it can be a lot more work to probe it. Suppose we have this contrived class C which we want to probe:

struct B {
  std::string s;
};

class C {
 public:
  C(int id) : id_(id) {}

  void add(std::string key, B b) {
    m_[key] = b;
  }

 private:
  std::map<std::string, B> m_;
  int id_;
};

It’s very complex, having STL data structures and other classes as member variables. Suppose we want to probe the member id_ when calling a function by passing C as reference:

__attribute__((noinline)) void random_func(C& c) {
  c.add("key", B{"hello 2"});
}

If we have debug symbols for the binary, we can inspect C’s structure to find the offset of id_:

$ pahole -F dwarf -C 'C' struct.cpp.dwo
...
class C {
        class {
        } m_;                                            /*     0     0 */

        /* XXX 48 bytes hole, try to pack */

        int                        id_;                  /*    48     4 */
public:
...
}

It tells us the offset of id_ is 48 bytes, so in bpftrace we can do:

uprobe:./struct:_Z11random_funcR1C
{
  $c = *(int*)(arg0 + 48);
  printf("x=%d\n", $c);
}

We can also add a pseudo-struct that mimics the offsets from C by declaring a struct inline in the script:

struct C_stub {
    char _pad[48];
    int x_;
}

uprobe:./struct:_Z11random_funcR1C
{
  $c = *(struct C_stub*)arg0;
  printf("x=%d\n", $c->x_);
}

which is slighly more readable and could be useful if multiple bpftrace scripts need this C definition.

The conclusion from my exploration on uprobes is that while they are provided out of the box, working with non-primitive types can get easily very complex and brittle. We now explore the scenario where we can modify the source code to make it easier to inspect.

Static Probes

We can add tracepoints to specific parts of the code and because it requires modifying the code, these are known as static probes. One of the easiest ways to do it is using the Folly library, via the folly/tracing/StaticTracepoint.h header, which defines a few macros. We can add those to one of our previous examples:

#include <folly/tracing/StaticTracepoint.h>

void my_print(const std::string& s) {
  FOLLY_SDT(my_project, start_proc, s.c_str());

  std::cout << s << std::endl;

  FOLLY_SDT(my_project, end_proc);
}
...

Suppose we compile this to strhist. To list the USDTs we can do:

$ sudo bpftrace -l 'usdt:./strhist:*'
usdt:./strhist:my_project:end_proc
usdt:./strhist:my_project:start_proc

The first thing to notice is that the prefix is now usdt instead of uprobe. The second is that because we provided the name specifically, the name is not mangled.

Also notice we passed s.c_str() which is of type char*, which is well supported by bpftrace. We don’t have to assume the data layout in std::string anymore, so our script becomes:

// strhist.bt
usdt:./strhist:my_project:start_proc
{
  printf("str=%s\n", str(arg0));
}

In this case s.c_str() is cheap enough to run every time, even when no tracing is taking place. If we want to gate so that code only runs during tracing, we can use FOLLY_SDT_IS_ENABLED + FOLLY_SDT_WITH_SEMAPHORE:

#include <folly/tracing/StaticTracepoint.h>

FOLLY_SDT_DEFINE_SEMAPHORE(my_project, start_proc);

void my_print(const std::string& s) {
  if (FOLLY_SDT_IS_ENABLED(my_project, start_proc)) {
    // This block is only executed when tracing is going on.
    FOLLY_SDT_WITH_SEMAPHORE(my_project, start_proc, msg.c_str());
  }
  std::cout << s << std::endl;
}
...

So USDTs are easier to work with but this requires more foresight.

Applications

Now that we know how to define tracepoints, we can do other types of analysis besides counting how many times a tracepoint is hit.

Latency

We can measure the amount of time it takes between two probes, say start_proc and end_proc. As long a end_proc tracepoint is always executed after a start_proc for a given thread, we can do:

usdt:./my_binary:my_project:start_proc {
  @ts[tid] = nsecs;
}
usdt:./my_binary:my_project:end_proc {
  @lat_us = hist((nsecs - @ts[tid]) / 1000);
  delete(@ts[tid]);
}

Upon termination, @lat_us will display a histogram of duration of end_proc - start_proc in microseconds.

Memory Allocation

We can measure how much memory was allocated between two probes, say start_proc and end_proc. As long a end_proc tracepoint is always executed after a start_proc for a given thread, and that we assume a specific memory allocator. Suppose the allocator is the standard malloc provided by libc. We can do:

uprobe:/usr/lib/libc.so.6:malloc
/ @in_req[tid] / { @req_size[tid] = sum(arg0); }

usdt:/my_binary:my_project:start_proc { @in_req[tid] = 1; }
usdt:/my_binary:my_project:end_proc {
  printf("allocs=%d\n", @req_size[tid]);
  delete(@req_size[tid]);
  delete(@in_req[tid]);
}

When we hit the first tracepoint, start_proc, we set a flag. If this flag is set, the malloc action will run, which in this case adds up the first argument.

This mechanism of setting a flag on start and clearing it at the end can be used with many other analysis described in [1].

Conclusion

I haven’t had the opportunity to use bpftrace in production at work, but now I have a better idea on what are the capabilities for a C++ application. I may update this post with other use cases should I run into them.

References

• [1] BPF Performance Tools - Brendan Gregg

Suppose we’re given a circle of radius $r$ and two points in its perimeter, and we want to compute the length of the arc between these points. This can be computed using elementary functions, for example by determining the angle $\theta$ in radians between these two points with respect to the center and then the arc length is $\theta r$.

For ellipses this is not as trivial. In the 18th century, Giulio Fagnano and Leonhard Euler were the first to use integrals to compute the arc length of an ellipse and these became known as elliptic integrals.

Niels Abel and Carl Jacobi studied the inverse of elliptic integrals and later realized they were doubly periodic, that is they have two fundamental periods, as oppose to trigonometric functions line sine that has a single period. Due to this connection double periodic functions became known as elliptic functions.

In this post we’ll study elliptic functions. We’ll start with simply periodic functions, then doubly periodic functions. We’ll spend most of the time studying properties of the periods.

Simply Periodic Functions

As the name suggests, periodic functions are those with image that repeats indefinitely, for example, $\sin z$, which repeats every $2 \pi$, that is $\sin z = \sin (z + 2\pi)$.

More generally, a function $f(z)$ is called periodic if $f(z) = f(z + w)$ for all $z$ in the domain, in which case $w$ is called the period. For this post, we’ll restrict ourselves to meromorphic functions.

If $w$ is the period of a function, that is $f(z) = f(z + w)$, inductively we can conclude that any integral multiple of $w$ is also a period, that is $f(z) = f(z + nw)$. If a period is not an integral multiple of any other period we call it the fundamental period. If a function has only one fundamental period, then $f$ is called simply periodic. For the remainder of the post, when we say periodic, we’ll imply simply periodic.

The function $f(z) = e^{2\pi i z / w}$ is the simplest periodic function and surprisingly, every periodic function can be expressed in terms of this one.

Lemma 1. Let $f(z)$ be a meromorphic function with domain $\Omega$ and period $w$. Then there exists a unique meromorphic function $g$ such that:

Discrete Fourier Series

Let $\Omega’$ be the domain of the function $g(\zeta)$ as defined in Lemma 1. Suppose it contains an annulus $r_1 \lt \abs{\zeta} \lt r_2$ in which $g$ has no poles. We can then express it via its Laurent series at 0 [2] as:

which then lets us express $f(z)$ as:

which is the Fourier series [3].

Properties

We now consider some properties of periodic functions. First, Lemma 2 shows periodic functions are closed under arithmetic operations.

Lemma 2. Let $f$ and $g$ be periodic functions of same period $w$. Then $f \pm g$, $f * g$ and $f / g$ are also periodic with period $w$.

Lemma 3. Let $f$ be a periodic function with period $w$. Then $f’$ is also periodic with period $w$.

Doubly Periodic Functions

As the name suggests doubly periodic functions have two fundamental periods. Doubly periodic functions are also known as elliptic functions, which is the term we’ll use henceforth. Any period $w$ of an elliptic function $f(z)$ can be written as a integral linear combination of any of its fundamental periods, denoted by $w_1$ and $w_2$:

We define the set of such periods as the period module of $f(z)$, denoted by $M$. Even if we don’t know $w_1$ and $w_2$ explicitly, we can guarantee that any integral linear combination of any other 2 periods in $M$ is also in $M$, as shown in Lemma 4.

Lemma 4. Let $M$ be the period module of $f(z)$ and $u, v$ any periods in $M$. Then any integral linear combination of $u, v$ is also in $M$.

As Lemma 5 shows, we can choose $w_1$ and $w_2$ such that $w_2 / w_1$ is not real, that is it has a non-zero imaginary part.

Lemma 5. Let $M$ be the period module of $f(z)$. Then there exists $w_1, w_2 \in M$ such that for all $w \in M$, $w = n_1 w_1 + n_2 w_2$ and $w_2 / w_1$ is not real.

Any pair of periods that satisfy Lemma 5 will be defined as a base for the module period.

Unimodular Transformations

Suppose we have a base $(w_1, w_2)$ and we want to perform a change of base to $(w’_1, w’_2)$. Since both are in $M$, each can be expressed as an integer linear combination of $(w_1, w_2)$:

In matricial form it becomes:

So we can convert from one base to another via a transformation represented by an integer matrix. A unimodular matrix is one where the determinant has modulus 1. Lemma 6 shows that the base transformation is a unimodular matrix.

Lemma 6. Let $(w_1, w_2)$ and $(w’_1, w’_2)$ be two basis. Let $T$ be the transformation that takes $(w_1, w_2)$ to $(w’_1, w’_2)$. Then $T$ is unimodular.

We can express the ratio of two bases as:

Dividing both sides of the fraction by $w_2$:

and from Lemma 6, we have $\abs{ad - bc} = 1$.

Canonical Basis

So far we’ve seen there can be multiple basis for a given period module. However, if we restrict the domain of the ratio $r = w_2/w_1$ for the basis, we can make sure it’s unique. The region we’re interested in is defined by the following constraints:

We call it the fundamental region and depicted it in Figure 1. Our first result is to show a base exists in it, as shown in Lemma 7.

Lemma 7. There exists a base $(w_1, w_2)$ for a period module with $r = w_2/w_1$ in the fundamental region.

Now that we know such a base always exists, Lemma 8 shows its ratio is unique.

Lemma 8. Let $(w_1, w_2)$ be a base for a period module with $r = w_2/w_1$ in the fundamental region, then $r$ is unique.

we define any base $(w_1, w_2)$ with $r = w_2 / w_1$ in the fundamental region as a canonical basis. Note that these are not unique given $(-w_1, -w_2)$ is a base with ratio $r$.

Lattice

Any base $(w_1, w_2)$ of the module period induces a lattice. The points on this lattice are those of the form $n_1 w_1 + n_2 w_2$. Each “cell” on this lattice is a parallelogram formed by adjacent points $(n_1 w_1, n_2 w_2)$, $((n_1 + 1) w_1, n_2 w_2)$, $(n_1 w_1, (n_2 + 1) w_2)$, $((n_1 + 1) w_1, (n_2 + 1) w_2)$.

The property of elliptic functions is that for any $a$, $f(a) = f(a + n_1 w_1 + n_2 w_2)$, so for any two points in different parallelograms in the same relative position within their respective parallelograms have the same value. Thus, an elliptic function is fully specified for the entire complex plane just from a single parallelogram.

Congruence Classes

We say that $z_1$ is congruent to $z_2$ with respect to $M$, denoted by $z_1 \equiv z_2 \pmod M$, if the difference $z_1 - z_2$ belongs to $M$. This induces congruence classes, and for $z_1, z_2$ on the same class $f(z_1) = f(z_2)$.

Properties

We now consider some properties of elliptic functions. Lemma 2 and Lemma 3 apply to elliptic function since they’re doubly periodic and satisfy:

By definition elliptic functions are meromorphic. If they have no poles then they’re holomorphics. Since $f$ is fully specified from a single parallelogram, its value is bounded in this finite region. According to Liouville’s theorem, a bounded holomorphic function must be constant.

Corollary 9. An elliptic function without poles is a constant.

If a function does have poles, Lemma 10 shows the sum of their residues is 0.

Lemma 10. The sum of residues of an elliptic function is 0.

Because simple poles (poles of order 1) have non-zero residue, it means that a non-constant elliptic function cannot have a single simple pole.

Lemma 11. A non-constant elliptic function has equally many poles as it has zeros.

An interesting consequence of Lemma 11 is the following. Suppose $f(z)$ has $N$ poles. If $p$ is a pole of $f(z)$, then $p$ is also a pole of $f(z) - c$ because if $1/f(p) = 0$ then $1/(f(p) - c) = 0$. From Lemma 11, $f(z) - c$ has the same number of zeros as $f(z)$, so there are $N$ values of $z$ which makes $f(z) = c$. Note that it doesn’t mean that the number of distinct zeros are the same for all $c$, since $N$ accounts for the multiplicity of zeros. The value $N$ is also known as the order of $f$.

In Alfhor’s [1] he claims that the order is the number of incongruent roots of the equation $f(z) = c$. This seems to imply it’s the same no matter $c$ and equals to $N$, but I don’t understand why this is true.

Theorem 12. Let $a_1, \dots, a_n$ be the zeros and $b_1, \dots, b_n$ be the poles of an elliptic function $f$ with module period $M$. Then

Conclusion

Elliptic functions are pretty interesting! I found the name pretty deceptive and more complicated-sounding than it actually is. I much prefer the term doubly-periodic functions.

I’m starting to see why elliptic functions appear in number theory, given that a lot of the results around the period module guarantee that the results are integers. This connection is fascinating!

Appendix

Lemma 13. Let $M$ be the period module of an elliptic function $f(z)$. There are either 2, 4, and 6 points in $M$ that are the closest to the origin.

Related Posts

Totally Unimodular Matrices. I was initially surprised to see totally unimodular matrices mentioned in the context of elliptic functions. I had studied them in the context of integer linear programming, but in hindsight it makes sense. There seems to be a deep connection between elliptic functions and integers and mentioned in the conclusion.

References

• [1] Complex Analysis - Lars V. Ahlfors
• [2] NP-Incompleteness - Zeros and Poles
• [3] NP-Incompleteness - Discrete Fourier Transforms
• [4] Complex Numbers Cheat Sheet
• [5] NP-Incompleteness - The Residue Theorem

In this post I’ll share my notes on the book Kafka: The Definitive Guide. Real-Time Data and Stream Processing at Scale by Gwen Shapira, Todd Palino, Rajini Sivaram and Krit Petty.

This book covers many aspects of the popular open-source Kafka, a distributed queue.

Book Summary

I read the 2nd edition of the book. It has 457 pages including Appendix, divided in 14 chapters. I skimmed most of the code snippets and chapters explaining configuration parameters since I was mostly interested in Kafka’s high-level architecture (I don’t use Kafka day-to-day).

Architecture. Chapter 1 provides a high-level overview of Kafka. Chapter 3 and 4 dives into Producers and Consumers, respectively. Chapter 6 dives into the core of Kafka.

Features. In terms of feature set (functional and non-functional) and applications, Chapter 7 describes Kafka’s reliability, and Chapter 8 explains how to achieve exactly-once semantics. Chapter 9 explains how to move data from/to Kafka to/from some other database, while Chapter 14 explains stream processing, i.e. how to transform data from Kafka.

Operations. Chapter 2 explains how to install it, including dependencies such as Zookeeper. Chapter 5 explains Kafka how to manage operations (e.g. creating topics) programmatically. Chapter 12 covers similar ground, but more manually, via CLI. Chapter 10 explains how to replicate data across data centers, Chapter 11 how to secure Kafka data and Chapter 13 how to monitor Kafka.

As I said, I was mostly looking into learning about how Kafka works. So the most useful chapters for my needs were Chapters 1, 3, 4, 6, 7, 8 and 14. These are the ones covered in my notes.

Kafka Overview

At the most basic level, Kafka is a distributed queue: you can have multiple hosts write data to it (producers) and have multiple hosts read from it (consumers) in a FIFO manner. One key feature from Kafka is that reading the data doesn’t remove the data from the queue. This allows multiple consumers to read the same data and even replay old data as long as its within Kafka’s retention.

Messages and Batch. A message is equivalent to a row in a database. For performance reasons, the unit of data in Kafka is a batch of data. How a batch is encoded is transparent to Kafka (of course it must be consistent between producers and consumers) but a typical serialization used is Apache Avro.

Relatedly, data can have an underlying schema, but these are also transparent to Kafka. The schema must be stored externally to be used by both producers and consumers.

Topics, Partitions and Segments. Kafka is actually a set of queues, not a single one. Each queue is called a topic.

For each topic there’s a corresponding set of N partitions. Each partition is replicated in one or more host (broker). The broker that contains the “master” replica is called the leader, while those containing the “slave” replicas are called the followers.

Note that a partition is a logical concept. At a broker level, it’s implemented by a set of files called segments.

Broker and Cluster. Is synonym with a host, but refers specifically to hosts holding partitions (producers and consumers are not brokers). Note that a host stores partitions from multiple topics.

Cluster is a logical grouping of brokers. In practice it seems like a cluster corresponds to a cluster deployment on a datacenter. The key constraint is that communication between brokers in a cluster must be fast.

Cross-cluster replication is possible, but it’s not used in the production path.

Consumers and Producers. Producers are a set of client hosts that write data to a specific topic. Multiple producers can write to the same topic.

Consumers read data from a topic. In general there’s too much data in a topic for a single host to be able to process it, so multiple consumers must be used to read the data in parallel. These form a consumer group, in which each member reads a unique portion of the data from a topic. Multiple consumer groups can read data from the same topic.

Checkpointing. Consumers may crash, the consumer group might scale up or down, so Kafka must know exactly up to which point a given consumer read its data. This is known as a checkpoint.

Since multiple consumer groups can read data from a topic, Kafka needs to store multiple checkpoints. This is stored colocated with the actual data in the broker.

Compaction. Kafka has the feature where messages can define a key (usually a column of the message) and only the last message of a given key is kept. This makes is kind of a key-value store, but the main purpose of this is to save space, so if a key is not needed, space can be reclaimed.

Architecture

Writing Data

The client wanting to write to Kafka must use a Kafka producer library. In the producer configuration, the user must provide the server address of at least one broker. This broker serves as a initial point of contact to return metadata about the other brokers in the cluster. With this information, the producer can talk directly to brokers it wants to write data to. There’s no intermediate gateway of sorts.

Partitioning. The producer requires the user to provide a key column which it will use to determine to which broker it should send the data to. This means the assignment is sticky: messages with the same key always go to the same partition. This also means it’s subject to hotkeys: if all keys are the same, only one broker will receive all the messages.

Batching. This producer doesn’t send data right away, it batches it for performance. Should the sending fail there are retry mechanisms provided. The user is responsible for handling client failures. If no persistence mechanism exists, if the client crashes the buffered data is lost.

Note within a batch all messages in a batch must belong to the same partition.

Serialization. The producer handles serialization from user format to bytes and the user can provide a custom serializer. It attaches a piece of metadata describing the schema of the data. It is not the full schema because the overhead would be prohibitive but rather a schema ID. It assumes the schema can be looked up from this ID. This enables non-back compatible schema evolution: if a producer serializes with schema ID X, and then later starts to write with schema ID Y, the consumer will know to deserialize the data with schema X or Y.

Reading Data

The client wanting to read from Kafka must use a Kafka consumer library. As mentioned before, each client is a member of a consumer group. It’s possible to scale in or out a consumer group by removing or adding more clients to the group. The consumer library will know how to re-distribute reads to them.

Sometimes we need key affinity, for example if the client aggregates the data and needs all the data with the same key to go to it. It’s possible to configure static assignment.

The way it works is that the consumer is by staying in an infinite loop and constantly calling poll() on the consumer client.

while (true) {
  ConsumerRecords<String, String> records = consumer.poll(timeout);

  for (ConsumerRecord<String, String> record : records) {
    int updatedCount = 1;
    if (custCountryMap.containsKey(record.value())) {
      updatedCount = custCountryMap.get(record.value()) + 1;
    }
    custCountryMap.put(record.value(), updatedCount);
    JSONObject json = new JSONObject(custCountryMap) ;
    System.out.println(json.toString());
  }
}

Checkpoints. As discussed earlier, Kafka uses checkpoints to know at which point of the queue a given consumer is reading. By default, Kafka commits a checkpoint for the data returned by a poll() when the next poll() is called. Assuming you don’t call the next poll() until you make sure the data is committed to the sink, this guarantees at-least-once semantics.

The consumer also has the flexibility to decide when Kafka saves a checkpoint, by turning off automatic commits and calling an explicit method, commitSync(). There’s an async version commitAsync() which doesn’t block during checkpoint but also doesn’t handle retries, so some care must be taken to ensure the right semantics. Exactly-once-semantics will be covered later.

Some care must also be taken when repartitioning happens. Suppose we add a new member to the consumer group. Then some existing members will “lose” the partitions they’re reading from, so they must commit a checkpoint before that happens. It’s possible to subscribe to such events.

The broker that keeps tab of offsets is called the group coordinator, and it is also the partition leader of a topic called __consumer_offsets (partitioned by the consumer group ID). This topic is only used for durability in case the leader crashes and the offset map must be reconstructed.

Cluster Membership

Kafka uses Zookeeper as a source of truth to which brokers are part of a cluster. Brokers must periodically send a heartbeat to the Zookeeper ensemble otherwise they’re considered dead.

Zookeeper is also used to elect a leader among the brokers, called the controller. The controller is responsible for deciding which brokers are the leader of a partition and which are the followers, so it must know when brokers leave or join the cluster.

Replication

The replication factor Kafka suggests is 2-3 but not more than that. Writes happen to the leader replica and typically reads are also only from the leader, but the latter can be changed to improve performance. The problem is that Kafka uses eventual consistency, so the replicas are usually not in sync.

The follower constantly asks for data from the leader, so the leader has an idea on how far behind each follower is. Followers lagging more than a configured time (10s by default) behind are considered out of sync, otherwise they’re in-sync replicas. On leader failure, only in-sync replicas are candidates for leader election.

Retention

Each partition is implemented via a set of files. By default each file contains either 1 GB or 1 week worth of data, whichever is smaller. When either of these limits is reached, Kafka starts writing to a new file.

The file contains metadata of the date range contained in it, so an asynchronous process can constantly delete files where all messages in it are out of retention.

Indexes

Kafka stores an index that maps logical topic offsets to actual files and file offsets. By default the granularity of the index is 4KB, i.e. for every 4KB of data written a entry is added to the index. This index also allows lookup by timestamp.

An index is a tradeoff between storage and compute. If we store every single message in the index it would take too much space. No index would make lookups take too long.

This index is needed because consumers can specify a time in the past for replay or to recover from a checkpoint.

Compaction

To implement the compaction, a background thread process a segment. It builds a in-memory hash table indexed by keys, containing the offset of the most recent message for that key. Then it does a second pass and it filters out any message that has a corresponding entry in the hash map with a smaller offset, and creates a new segment. The book is very confusing about this process, suggesting this is done in one pass.

Note that this is an “eventual” compaction. Consumers reading from the past might still see “dirty” messages. It’s assumed that clients reading from compacted topics are themselves writing to some sort of key value store that also retains only the last entry.

A similar mechanism is used even for non-compacted topics, when data must be deleted. The producer might indicate a message called a tombstone, which is basically a message with the key to be deleted and a null value. Consumers are supposed to handle this tombstone: if they process this message

Exactly-Once Semantics

By default, Kafka offers at-least-once semantics. This means that on consumer restart or repartitions, it guarantees that all data it stores will be sent to the consumer but it might send the same data more than once.

Achieving exactly-once is very difficult because the process of reading, processing and writing data must become atomic, so Kafka must be involved in the whole process. Since Kafka must be aware of the writes, a limitation of this process is that the sink of the application must also be a Kafka topic.

Chapter 8 describes the setup necessary.

Idempotent Producers

To start, the producer must be made idempotent. This means that each message will contain extra metadata: the producer ID and a sequence number, which will be used to uniquely identify a message and identify gaps. For each partition, the broker will keep the last N messages from each producer. If it gets a sequence already in this N messages, it will simply deduplicate.

If it gets a sequence number out of that range it will fail. If the last sequence number it has is x, and it gets x + 2, it will detect a gap in the sequence and fail as well.

Transaction

The transaction Kafka implements is by handling both the writes to the sink and the checkpointing. The producer writing to the sink must be marked as transactional. Here’s an example of a write with transactions (compare this with the simpler example in Reading Data):

producer.initTransactions();

while (true) {
  try {
    ConsumerRecords<Integer, String> records = consumer.poll();
    if (records.count() > 0) {
      producer.beginTransaction();
      for (ConsumerRecord<Integer, String> record: records) {
        ProducerRecord<Integer, String> customizedRecord = transform(record);
        producer.send(customizedRecord);
      }
      Map<TopicPartition, OffsetAndMetadata> offsets = consumerOffsets();
      producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
      producer.commitTransaction();
    }
  } catch (KafkaException e) {
    producer.abortTransaction();
    resetToLastCommittedPositions(consumer);
  }
}

Init Transactions

To prevent multiple producers with the same ID writing to Kafka, it uses a fencing mechanism. It stores a map of producer ID -> timestamp. When a producer calls initTransactions() it sets the current timestamp for its ID in that map. It will keep using this same timestamp when writing messages.

When writing a message, the broker will check if the timestamp on the message matches the one in the map. If not it will reject. This prevents multiple producers from being alive at the same time. If a new producer starts and registers a higher timestamp, it overtakes all the other existing producers.

When a producer calls initTransactions() Kafka also selects one of the brokers to be the transaction coordinator (hash of the producer ID, so the mapping is sticky). It will keep in-memory metadata about the transaction, such as state and which partitions are participating in the transaction. This broker is also the partition leader of a special internal topic named __transaction_state (partitioned by producer ID) which can be used to restore the state.

Begin Transaction

To start a transaction, the producer calls producer.beginTransaction(). This doesn’t talk to Kafka, but the producer client is aware that the next message it sends will be inside a transaction and it will tell Kafka that.

When the producer does send a message the coordinator will first add a message in the topic __transaction_state indicating a transaction started. The message is then processed by the respective brokers as normal, but they mark the message as uncommitted (so that doesn’t get returned to consumers).

The coordinator writes a message to __transaction_state with all partitions that have changed by the current write.

In addition to sending the messages, we also need to update the offsets for the consumers. Note that it’s the producer that commits the offset now, not the consumer: producer.sendoffsetsToTransaction(offsets, consumer.groupMetadata()).

The group coordinator will keep non-committed offsets in a special place and once the transaction is committed it will merge these offsets into its main map.

Commit Transaction

When the producer calls producer.commitTransaction(), the coordinator writes a message in the topic __transaction_state with the state PREPARE_COMMIT. Then it sends a request to the leader of all partitions to add a special message representing a “commit” marker. So when broker returns its messages, only those preceding the last “commit” marker are considered valid.

Since the offset topic __consumer_offsets has also been changed during the transaction, it’s also notified and will add the “commit” marker.

Abort Transaction

Similarly, if the transaction needs to aborted, the producer calls producer.abortTransaction() and the coordinator go through the same process but instead of a “commit” marker, it will have a “abort” marker. Messages between starting at a marker and ending in the “abort” marker are considered deleted.

Differently from the success case, in an aborted transaction, we need the consumer to call resetToLastCommittedPositions(). While the __consumer_offsets also received the “abort” marker and will know not to return the pending offsets, on the client side we already moved to the next offset, so need to reset.

Data Pipelines

Kafka is the input source for stream processing frameworks such as Flink, but it also provides its own stream processing framework known as Kafka Connect. This is mostly useful for sending data from Kafka to another source type (e.g. MySQL) or vice-versa, with simple transformations.

Related Posts

In the post Queues we discussed queues in general.

Conclusion

I got what I wanted from this book: I learned a lot about Kafka! The part I found the most interesting and difficult was the transactions. I found it amusing how Kafka uses special topics for a bunch of internal operations including checkpointing and transactions, but also cluster-replication. It’s how in Linux everything is a file and in Kafka everything is a topic.

As I was trying to write down my understanding, I found some difficult topics such as compaction and transactions were missing details so I had to complement with external research.

As I tried to summarize the chapters I realized that related chapters don’t seem to be grouped together and some like Chapter 5 and 12 feel like they should be one. This seems to be a common artifact of multi-author books, but didn’t impact the content of the book.

This is a meta-post to review what happened in 2025. Every year I go over the posts I wrote, reflect on the blog as a whole and on the personal side, share things I've done (mostly trips and books read). The thumbnail was generated by "Your year with ChatGPT" and found it reflects well what I share in this post.

Posts Summary

In 2025 I focused mostly in my studies of Complex Analysis, C++ and operating systems. I was happy with the balance of work vs non-work related (almost half-half).

Complex Analysis

I started reading the book Complex Analysis by Ahlfors in September of 2023. I’ve made steady progress on it in 2025 but I’ve changed my goal of covering every single topic on the book. I skipped a few topics from the chapters on the Dirichlet problem.

My current plan is to study the elliptic functions chapter but skip the last chapter on globally analytic functions and my new estimate is that I can finish this book before mid 2026.

Here’s a list of all the posts I wrote as notes from the book this year:

• The Maximum Principle
• The General Form of Cauchy’s Theorem
• The Residue Theorem
• Runge’s Theorem
• Mittag-Leffler’s Theorem
• Weierstrass Factorization Theorem
• The Gamma Function
• Harmonic Functions
• Hadamard Factorization Theorem
• The Riemann Zeta Function
• Subharmonic Functions
• The Perron Method

Work

I learned more about C++ such as Views, Concepts and CPOs. I also studied Folly futures, executors, coroutines and their lifetimes. As a background for Folly executors I also studied asynchronous I/O.

I studied how C++ binaries are compiled (in particular by LLVM), linked (via shared libraries) and executed (ELF and jemalloc). More generally, I also had posts on distributed systems and operating systems: I wrote about queues, read books on microservices, system performance and BPF.

I’m satisfied about being able to read so many technical books this year. I did this by reserving 30-60 min each morning to reading, and got through 4 books amounting to a few thousands of pages. I applied some techniques described in Atomic Habits: I comitted to read at least one page a day and more importantly I started using a new phone without any work related nor social media apps. It’s incredible how much time I had been wasting each morning on those.

Leisure

I didn’t explore much outside of work and complex analysis but I enjoyed writing about my last visit to the Computer History Museum.

I belatedly completed the Advent of Code 2024. It was a fun experience and I started writing a post about the problems but didn’t have time to finish.

Personal

I wrote more “shower-thoughts” / personal opinion posts about neomania, horseshoe theory and complex systems. A bit short of my plan of writing once a quarter but I’ve enjoyed working on these and hope to continue next year.

The Blog in 2025

Numbers

I’ve have a resolution to post at least once a month on average but this year I actually averaged more than one post every other week, by writing 33 of them. The blog completed 15 years with 269 posts.

I still use GoatCounter for analytics. The tool A Bulls and Cows Solver continues to be the most visited page with 4,744 visits. This is a solver I built for fun in 2018 as part of a post, Bulls and Cows, which is a code-breaking 2-player game. I explain why this page in popular in my 2024 review.

Restricted to the blog, Linear Predictive Coding in Python written in 2021 continues to be the most popular post with 654 visits. Surprisingly, a post I wrote in 2024, Understanding std::call_once() in C++ became quite popular with 571 visits. From this year, Folly Executors had the most visits, 119.

Overall traffic to my website grew by a tiny amount, from 17,098 to 17,145.

AI

I’ve been noticing a drop in traffic to my blog posts, which I attribute to tools like ChatGPT and the fact that Google now often answers questions inline without users having to click links.

I actually don’t mind this change in pattern too much. I never wrote with visits in mind, but I’m happy when I see a popular post because I assume people find it useful. With AI it’s still possible that my posts are helping people behind LLMs but I lost any visibility.

Citations

From time to time I Google my blog address to see if I can find any interesting references and this time I found it has been cited by a paper by folks at Purdue: Unbalanced optimal transport for stochastic particle tracking . I couldn’t find it in any journal and it doesn’t seem peer reviewed, but it cites a 2013 post Totally Unimodular Matrix Recognition.

Most of my posts do not provide original ideas, being just notes on books and papers I read, so I’m surprised that anyone would cite it instead of the original source directly, but then most textbooks are also not original (in content, but are in presentation) often sourcing from papers and still, many papers cite textbooks.

Resolutions for 2026

As I mentioned above, I’ll commit to finish learning about complex analysis. I’m hoping to have time to start a new topic, which for now is classical mechanics.

I also want to explore more LLM tools such as Claude. The landscape has been changing very fast and I find hard to be on top of the best tools, so I want to dedicate time to explore them in more depth. I usually don’t follow the newest trends but this might time well-spent because it can make me more efficient and end up saving time in the long run.

Personal

The end of the year is a good time to look back and remember all the things I’ve done besides work and the technical blog.

Trips

In February, we visited Singapore and Vietnam. It was my first time in Southeast Asia and I learned a lot about the rich Vietnamese history and enjoyed the local food a lot.

In November after a business trip to London, we decided to visit Morocco. I also learned a lot about Morocco’s history and was fascinated by its Islamic architecture, especially the tile work (zelije). It was a nice historical connection to the trip to Andalucia last year because the Moors from Andalucia came from Morocco.

We also did quick trips to Portland, Oregon and Denver and the Rocky Mountain National Park in Colorado.

Books

This year I started publishing the notes for books I read as soon as I finish them. In fact from now on I don’t consider a book read until I do that. The list of all books I read this year is now on my books page. In this post I’ll make some overall commentary.

Fiction and Poetry

This year I read more fiction than usual, in particular sci-fi. Recommended by a friend, Manna is an interesting thought-experiment as to what can happen if most of human work is automated. I’ve been wanting to read Asimov for a while and finally scratched the itch with I, Robot. It contains lots of interesting ideas between human-robot interactions.

I initially didn’t enjoy reading Borges’ Ficciones because I couldn’t grok the heavily philosophical first story but the subsequent ones were pretty interesting. Also, after reading the entries on Wikipedia I realized there were many other layers I had missed. I especially liked the Tower of Babel.

Other fictions I read were The Tempest and Pachinko. The Tempest was the first text I read from Shakespeare and found it underwhelming. Pachinko was okay.

I read my first book in Spanish, Romancero Gitano, a poetry book with very involved metaphors. It would have been hard to understand this book even in my mother language, but luckily it has lots of comments.

History and Travel

I finished reading The Scramble for Africa which I started in 2024 after visiting a few countries in Southern Africa. It covers a short period of time but geographically it’s very broad: it mentions a large part of countries in Africa.

For trip-related books, this year I read Perfume Dreams and Fire in the Lake for Vietnam and Singapore: A Very Short History for Singapore. While they’re not bad, I didn’t get the historical information I was after, except for the Singapore one. I decided that going forward I’ll just read Wikipedia.

I did just this for Morocco. I read the main Wikipedia page and found it a lot more effective at learning about its history. To complement it, I read the memoir For Bread Alone by the Moroccan Mohamed Choukri in hope of some cultural immersion, without success. I still haven’t found a good way to learn about a country’s culture through books. The closest I got is through travel guides like Lonely planet.

With similar hopes I read the book A Lady’s Life in the Rocky Mountains for a trip to Colorado and Portlandness for Portland. The former wasn’t very insightful, but the latter did provide some interesting cultural background to Portland and I enjoyed the data viz from it.

STEM

I read The Body which covers interesting bits about the human body. I also re-read The Music of the Primes which I had read over a decade ago. I knew it was about the Riemann Hypothesis, and I wanted to revisit it after studying The Riemann Zeta Function .

I also read the biography of von Neumann, The Man from the Future. I like learning about the history of mathematics, so I wanted to see if I’d enjoy reading biographies of mathematicians. It was an interesting but not an amazing read, so not sure I want to make a habit of it.

Psychology

I enjoyed reading some books on self-improvement such as The Almanack by Naval Ravikant, Nonviolent Communication and Atomic Habits (re-read).