td { padding-right: 10px; }

Title slide reading'Dynamic Modules: How They Work' for nginx.conf 2016 presentation by Maxim Dounin, developer of dynamic modules at NGINX, Inc.

This post is adapted from a presentation delivered at nginx.conf 2016 by NGINX, Inc. developer Maxim Dounin. You can view a recording of the presentation on YouTube.


Maxim Dounin: My name is Maxim Dounin, and I am an NGINX developer. Today, I would like to talk about dynamic modules, which we introduced last winter.

0:17 The Basics

Background about NGINX dynamic modules: NGINX has always had a modular design, but previously modules could only be compiled in statically; dynamic modules were introduced in NGINX 1.9.11 after development by the speaker (Maxim Dounin) and Ruslan Ermilov

First, let’s review some basics about modules in general, and dynamic modules in particular.

As you probably know, modules in NGINX have existed since its inception. In fact, NGINX is actually a collection of modules. Even very basic functions like HTTP, or serving static files inside HTTP – these basic things are modules.

You can extend NGINX by adding your own modules. The modular architecture allows you to easily modify NGINX. I think it’s a very important part of NGINX. It’s what makes it so good, and so successful.

Last winter, we introduced dynamic modules. You can now load modules into a compiled NGINX binary, without recompiling NGINX itself. Dynamic modules were the result of a joint effort by myself and Ruslan Ermilov.

1:42 Are Dynamic Modules Really Needed?

A main motivation for implementing NGINX dynamic modules is to simplify packaging, especially for modules with external dependencies

Why are dynamic modules needed?

Normal static modules have existed for ages, and they work fine. Moreover, NGINX has a very useful and cool feature. You can change the binary in place, without dropping any requests. This feature is normally used for upgrades, but you can also recompile NGINX with a different set of modules and just switch to a different binary. Changing sets of modules is not a problem.

Packaging is a problem, though. It’s quite difficult to package NGINX with different modules, especially when there are external dependencies. You may have to ship NGINX without modules, or you may have to ship single?loaded packages. It’s awkward, and doesn’t always work very well.

3:00 Main Goals of Dynamic Modules

Another motivation for implementing NGINX dynamic modules is to simplify debugging: by commenting out a module, you can easily determine whether it is causing a particular bug

A main goal of creating dynamic modules was to simplify package management. Another goal of dynamic modules was to simplify debugging.

Users can just comment out third?party modules to check whether a bug they are experiencing was introduced by some third?party module that they compiled in accidentally.

3:32 Mini How?To

Slide summarizing how to compile dynamic modules: with the'=dynamic' flag for standard modules or the'--add-dynamic-module' option for third-party modules; also add the load_module directive in the NGINX configuration

Here’s a short how?to for using dynamic modules.

You can compile NGINX with several standard modules dynamically loaded, by using the =dynamic suffix. You can compile third?party modules dynamically by using the --add-dynamic-module option to configure. And then in NGINX configuration, you use the load_module directive to load a module.

4:07 Implementation Details

Introductory slide for the section about implementing dynamic modules in the nginx.conf 2016 presentation by Maxim Dounin, developer of dynamic modules at NGINX, Inc.

Let’s now take a look at some implementation details. We’ll look at some of the more interesting things we encountered while implementing dynamic modules.

Two principles guided the implementation of NGINX dynamic modules: simplicity and minimal required changes to existing modules

First of all, we decided that implementation should be simple.

There are a lot of high?value third?party modules and we can’t just force all developers to change their modules and make life hard for them. So, it was important to minimize required changes to modules.

4:45 Initial Trivial Solution

The first approach to implementing NGINX dynamic modules was to have NGINX read in all the modules located in a specific directory, but the load_modules directive was added instead

Our initial trivial solution was to just load all modules that exist in a certain directory when NGINX starts up. This requires no changes to module code, but it still required changes to module config scripts.

So, then we implemented something a bit more complex with the load_module directive in the configuration. Now you could load and unload modules from the configuration, and you didn’t need to rename files, or move files around.

5:29 Subtle Parts

Summary slide for section about the solutions chosen for challenges in the implementation of NGINX dynamic modules

These are some problems we were working on while implementing dynamic loading.

Let’s look at them in order.

5:46 Module Config Scripts

The'auto/module' configuration script was introduced to maintain compatibility between existing static modules and the new dynamic modules in NGINX

First, module config scripts. This was probably the most complex part from an implementation point of view.

There are many existing modules and compatibility is very important. You don’t want to break things. So, we had to provide compatibility with all modules.

Module config scripts are essentially shell code. In these config scripts, modules just set some global variables. NGINX configuration then recognizes these variables and uses them. We’ve left everything as is, so the same variables are still used by NGINX configuration.

We’ve introduced the auto/module script to assign these variables. So, now there’s an interface to assign these variables for static compilation, and it also knows how to assign variables for dynamic compilation. By converting your module config script to use the auto/module script, you make it clearer and also add dynamic compilation.

A bit later we’ll take a look at how to do this for a sample module. It’s quite trivial.

7:28 List of Modules: ngx_modules

Implementing NGINX dynamic modules required changing how the list of modules is set; the ngx_modules global variable used for static compilation was replaced with cycle-modules

The second problem regards a list of modules inside NGINX. It’s set with a global variable, ngx_modules.

The problem is that each configuration needs its own list of modules. Therefore, we have to replace the global variable ngx_modules with cycle->modules inside a list of modules, inside a particular configuration.

The ngx_modules global variable is still there. If a module doesn’t need to support dynamic loading, it can continue to use ngx_modules as is, but if a module is a complex one that needs to be converted to understand dynamic loading, it needs to be changed to use cycle->modules instead.

8:34 Module Indexes

Implementing NGINX dynamic modules required solving issues with module indexes; nginx_count_modules() was introduced to assign'ctx_indexes' for complex modules

Another problem is module indexes.

Each module is described by a global structure, ngx_module_t. This means that indexes that are stored within the structure cannot be changed. If the index is assigned, you can’t change it later.

Moreover, there must be no conflicts with previous configurations. If you’ve loaded a module in a previous configuration, and in the new configuration you haven’t loaded this module yet, you still can’t reuse the index of this module, because this module might be loaded later.

This is something the NGINX core takes care of while loading modules, and we’ve introduced the ngx_count_modules() function to assign ctx_indexes for complex modules.

9:48 Order of Modules

Implementing NGINX dynamic modules required adding code to use information recorded in a module to determine its place in the load order

The next issue is the order of modules.

The order is important in some cases, especially if you write a filter. For example, gzip must happen before writing the data to a client. Both these tasks are done by filters, so you have to specify the appropriate order for the filters to run in.

During static compilation, this is done via the configuration order. But when you’re loading a module, there is no [predetermined] order, so we have to store a configuration-imposed order inside the module itself. Then, when NGINX loads the module, it takes a look at the order recorded in the module, and puts the module in the appropriate place in the module list.

11:01 Signature Checking

Implementing NGINX dynamic modules required adding signature checking to prevent loading of modules compiled with a different set of configuration options than the binary

The next problem is that NGINX uses conditional compilation on various structures.

NGINX doesn’t compile some fields of the structures which are not used in a particular compilation. And there’s no API layer between these structures and the modules. The structures are essentially a part of our API. So, if you try to load a module that was compiled with a different set of configuration options, it will likely segfault because it assumes a different structural layout.

To prevent this, we’ve introduced signature checking.

NGINX checks if the module was compiled with the same options. If it was compiled with a different set of options, NGINX refuses to load the module. We have managed to implement this in an existing macro, so no changes to modules are required at all. It just seems to work like magic.

12:36 Version Checking

Implementing NGINX dynamic modules required version checking to guarantee that a dynamic module is compiled against the same NGINX version as the binary

A similar issue is version checking.

The same problem with structures happens when you switch from version to version: segfaults happen when loading a module compiled with a different NGINX version.

Therefore, while loading modules, NGINX checks the version the module was compiled with.

13:14 Filter Chains

A final challenge to implementing NGINX dynamic modules is to figure out how to handle filter chains, especially when a worker process terminates

The last problem is filter chains.

Filter chains are global like the list of modules. They’re initialized during post?configuration callbacks, so most configuration problems (like syntax errors, etc.) happen before this, and in that case the filter chains are not initialized.

But some errors can still happen after the filter chains are constructed. If a configuration fails after the filter chains are built, NGINX rolls back to the old configuration, and unloads the modules that were just loaded. As a result, the NGINX master process is in an inconsistent state. It does not have a module loaded, but the module is linked in the filter chain.

This is not fatal, as a master process doesn’t call filters itself. Normally, it will just change the configuration, build another filter chain, and will spawn new workers with correct configuration and filter chains.

But there’s one small case when this can be a problem. If a worker process dies, NGINX tries to respawn the worker with the current master state. This state is inconsistent, so the worker dies too because it is referencing a module that’s not loaded. This problem requires reimplementing filter chains in a way that is specific to each configuration.

Because this happens only in some very specific cases, we’ve left things like this; fixing it would require too many changes to modules. If you do have a good solution, please let us know.

16:00 What We Ended Up With

Slide summarizing the new code written for conversion from static module compilation to dynamically loaded modules, which are discussed in detail on subsequent slides [nginx.conf 2016 presentation by Maxim Dounin, developer of dynamic modules at NGINX, Inc.]

What we ended up with is the auto/module configure script, the ngx_modules global variable replaced with cycle->modules, the ngx_count_modules() function to assign context indexes for complex modules, and the load_module directive.

All previous modules continue to work fine, and most modules can be easily converted to dynamic loading with just a few simple changes.

16:41 Conversion

To convert a module from static compilation to dynamic loading, you need to rewrite the configuration script to use the'auto/module' script; additional steps are necessary for more complex modules

Let’s look at what’s needed to convert a module for dynamic loading. There are actually just three simple steps for all modules. And in most cases, only one is required.

In all cases, you have to rewrite your config script to use auto/module. In complex cases, you also have to use ngx_count_modules() and you have to replace ngx_modules with cycle->modules.

17:15 ngx_count_modules()

For some NGINX dynamic modules, to convert from static compilation you need to replace the existing loop function with ngx_count_modules()

Let’s start with the most complex case.

Here’s an example from NGINX itself: how to count modules. It basically replaces a loop in the code with a function call.

17:41 ngx_modules to cycle->modules

For some NGINX dynamic modules, to convert from static compilation you need to replace'ngx_modules' with'cycle-modules'

And here is how to replace ngx_modules with cycle->modules. Again, we just replaced the global references with cycle->modules.

17:57 Config Script

For most NGINX dynamic modules, all you have to do to convert from statically compiled modules is change your configuration script

In most cases, you have to rewrite your config script. In most cases, this is actually the only thing you have to do.

Here is an example for a simple module. The original version uses global variables. It specifies a domain, HTTP modules, and add?on sources.

The new version uses the auto/module script. It basically does the same thing as the original version but uses the auto/module script, which allows it to selectively specify the module type, name, included directories, dependencies, sources, and libraries.

That’s all you need to do in this case.

18:51 Compatible Config Version

The ngx_module_link variable in the'auto/module' script for loading NGINX dynamic modules enables compatibility between NGINX versions

In some cases, you may want to provide compatibility with previous versions of NGINX.

This can be easily done by testing for the ngx_module_link variable. If it’s set, you use the new version. If it’s not set, you use global variables as in previous versions.

19:20 Parameters of auto/module

The'auto/module' script for loading NGINX dynamic modules has several parameters for specifying the module type, its name, the include directories, dependencies, sources, and libraries, and other settings

Here’s a short list of parameters for the auto/module script. You can specify module type, usually HTTP or HTTP_AUX_FILTER, the name of the module, include directories, dependencies to check when building, sources, and libraries.

Some special values are recognized for libraries NGINX knows how to build. You can also just provide a link of flags like NGINX’s own test C++ module.

There are a couple more variables I haven’t showed yet. One is ngx_module_link, which specifies linking types; do set it directly. NGINX expects to set it itself before calling the config script.

The order of modules to be used is normally set based on module type. You don’t need to worry about it unless you are doing something specific.

20:39 Complex Modules

You can build'complex'; dynamic modules for NGINX that combine multiple modules into a single .so file

It is also possible to compile many modules into a single, shared object file.

You specify multiple module names and multiple sources with the --add-dynamic-module parameter to the configure script [for dynamic modules] or with the --add-module parameter [for the complete NGINX binary].

21:06 How Loading Works

Title slide for section about how loading of dynamic modules works, in nginx.conf 2016 presentation by Maxim Dounin, developer of dynamic modules at NGINX, Inc.

Now, let’s take a closer look at how module loading works, how it happens, and the results.

21:20 The load_module Directive

When NGINX reads in its configuration, the load_module directive calls dlopen(); it calls dlclose() when the configuration is freed

The load_module directive calls dlopen(). Later, when the configuration is freed, dlclose() is called.

21:35 Configuration Reload

The internal mechanism for loading in an updated version of a dynamic module in NGINX means there is not downtime and you can load modules in any order

What happens when you reload the configuration?

The new configuration calls dlopen(). The previous configuration is freed, and it calls dlclose(). If the module is in both the previous and the new configuration, it remains loaded.

This is a good thing, as it means you can even remove the module file itself, and NGINX doesn’t notice. You can still reload the configuration when needed.

It allows you to replace the NGINX binary and NGINX modules in any order. You can still reload the configuration if needed.

22:36 How to Upgrade a Module?

You can upgrade to a new version of a dynamic module on the fly, the same as for the binary itself, with the'service nginx upgrade' or'kill -USR2' command

If you want to upgrade a module without upgrading NGINX, you actually do the same thing you do when upgrading NGINX itself.

You use the USR2 signal, or on most systems, service nginx upgrade which sends the USR2 signal for you.

23:02 Further Improvements

At the time of this nginx.conf 2016 presentation by Maxim Dounin, developer of dynamic modules at NGINX, Inc., some planned compatibility improvements were not yet available; they were released in NGINX Plus R11

The last topic I wanted to talk about today is further improvements. These are things we are still working on and would like to share with you.

23:14 Compatibility Between Builds

The --with-compat option enables interoperation between dynamic modules and NGINX or NGINX Plus binaries built with different options to the'configure' command; introduced in NGINX Plus R11

First, compatibility between builds.

This is actually related to signature checking. It’s a pain to compile dynamic modules if you do it outside of NGINX itself. You have to use the same configure options, or signature checking prevents you from loading the module.

We are currently working to make this easier. With a special option to configure, perhaps something like --with-compat, NGINX will just compile in all the structure fields it knows about, so that modules will be compatible even with different configure options. You will be able to load them even if you compile NGINX and the module with a different set of configure options.

[Editor – The --with-compat option was introduced in NGINX Plus R11 and NGINX 1.11.5.]

24:27 Compatibility with NGINX Plus

Binary compatibility between dynamic modules and NGINX Plus was not available at the time of this talk, but was introduced in NGINX Plus R11 [nginx.conf 2016 presentation by Maxim Dounin, developer of dynamic modules at NGINX, Inc.]

Closely related to that is compatibility with NGINX Plus.

Right now, it is not possible to load a module compiled for NGINX into NGINX Plus, because of signature checking, and the incompatibilities between structures.

We are working to make that possible.

[Editor – Binary compatibility for NGINX Plus was introduced in NGINX Plus R11 and NGINX 1.11.5.]

24:54 Thank You

Final slide reading'Thank you' in nginx.conf 2016 presentation by Maxim Dounin, developer of dynamic modules at NGINX, Inc.

Thank you for your attention. If you have any further questions about dynamic modules, feel free to contact me.