Enabling OpenTracing with the NGINX Ingress Controller for Kubernetes

Interest in Kubernetes has grown steadily over the past few years, and many enterprises are adopting it along with microservices architectures in their production environments. Adopting distributed services introduces new challenges, however. Understanding and debugging what is happening in a microservices‑based application can be difficult, especially if there are a large number of services. To […]

Read More…

The post Enabling OpenTracing with the NGINX Ingress Controller for Kubernetes appeared first on NGINX.

Interest in Kubernetes has grown steadily over the past few years, and many enterprises are adopting it along with microservices architectures in their production environments. Adopting distributed services introduces new challenges, however. Understanding and debugging what is happening in a microservices‑based application can be difficult, especially if there are a large number of services. To spot failures or performance problems, you need a distributed tracing tool which tracks requests end-to-end as data is passed among the microservices that make up an application.

OpenTracing is a specification and set of APIs for distributed tracing. In a previous post we showed how to enable distributed tracing for applications proxied and load balanced by NGINX Open Source and NGINX Plus, using the open source module (nginx-opentracing) created by the OpenTracing community. At the time of writing, Open Tracing provides libraries for nine programming languages.

Now, we are also adding native support for OpenTracing of HTTP and gRPC requests in a Kubernetes cluster when using the NGINX and NGINX Plus Ingress Controllers for Kubernetes to load balance the traffic in the cluster.

There are a variety of use cases for using OpenTracing, and here we’re focusing on tracing server endpoints with request context propagation. In a distributed environment, each application inside the cluster is seen as a different server. Imagine two apps or services that are both involved in processing a request from a client. For example, in the topology shown in the following diagram App1 is a web server that handles an HTTP request and redirects it to App2. Both applications are running inside a Kubernetes cluster load balanced by the NGINX Ingress Controller and OpenTracing is enabled, so we can trace the request from the Ingress Controller as it passes through App1 and reaches App2.

Information about each operation performed by a component in the system (such as a service) is captured in what OpenTracing calls a span, and chaining the spans together enables us to identify and track a request during its journey through the microservices in the cluster.

Note: At the time of writing, OpenTracing is available only in the edge version of the NGINX Ingress Controller.

Building OpenTracing into the NGINX Ingress Controller Image

To use OpenTracing with our Ingress Controller, you need to incorporate the OpenTracing module into the Docker image for the NGINX or NGINX Plus Ingress Controller, and designate the tracer you’re using.
In the GitHub repository for the Ingress Controller, we provide separate Dockerfiles for NGINX and NGINX Plus. They both incorporate the open source OpenTracing module it into the Docker image, but in different ways:

  • For NGINX, DockerfileWithOpentracing downloads the OpenTracing module from GitHub and compiles it in manually during the first stage of the Docker build.
  • For NGINX Plus, DockerfileWithOpentracingForPlus uses the package manager to retrieve the dynamic module that NGINX builds from the open source OpenTracing module and maintains.

Perform these steps:

  1. (Optional.) Designate a tracer other than the default, Jaeger. Plug‑ins are also available for Datadog, LightStep, and Zipkin. To use a different tracer, modify the Dockerfile as specified in the Prerequisites section of our instructions for enabling OpenTracing.
  2. Build the Docker image using the instructions in the Ingress Controller repo. In Step 3, specify the appropriate Dockerfile.

    For NGINX:

    $ make clean $ make DOCKERFILE=DockerfileWithOpentracing PREFIX=YOUR-PRIVATE-REGISTRY/nginx-ingress

    For NGINX Plus:

    $ make clean $ make DOCKERFILE=DockerfileWithOpentracingForPlus PREFIX=YOUR-PRIVATE-REGISTRY/nginx-plus-ingress
  3. Install the image using the instructions in the Ingress Controller repo.

    Before running the kubectl apply command in Section 3 of the instructions, remember to update the YAML file to specify the newly built image that has OpenTracing incorporated:

    containers: - image: IMAGE_WITH_OPENTRACING
  4. Deploying the Jaeger Tracer

    For this blog we’re using the default tracer, Jaeger, and deploying it inside the cluster for simplicity, but it can be also deployed externally. The Ingress Controller pod and the application pods that use OpenTracing must have access to the tracer.

    Also for simplicity, we’re using the all-in-one template provided by Jaeger to set up a non‑production Jaeger instance in the cluster. The resulting deployment – suitable for a development environment – creates a Jaeger pod and the set of services needed to access the pod; it sets up the latest Jaeger version in the default namespace, with in‑memory storage and limited functionality.

    $ kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml

    In production, we recommend using the Jaeger Operator for installation. For more information about Jaeger, see the Jaeger website.

    Enabling OpenTracing

    The following ConfigMap (nginx-config.yaml) for the NGINX Ingress Controller enables OpenTracing globally. It adds three new ConfigMap keys in the data section:

    • The opentracing key enables OpenTracing for all Ingress resources created in the cluster.
    • The opentracing-tracer key specifies the path to the tracer library, which gets downloaded and copied into the Ingress Controller image when you build it.
    • The opentracing-tracer-config key embeds the tracer configuration. The service_name field defines the service associated with the actual span, and the reporter field specifies the address and port of the tracer (in our case, the address and port of the Jaeger service deployed by the all-in-one template in the previous section). The sampler field specifies the client sampling configuration; for simplicity, we’re configuring the “Constant” sampler (const), which samples all traces.
    kind: ConfigMap apiVersion: v1 metadata: name: nginx-config namespace: nginx-ingress data: opentracing: "True" opentracing-tracer: "/usr/local/lib/libjaegertracing_plugin.so" opentracing-tracer-config: | { "service_name": "nginx-ingress", "sampler": { "type": "const", "param": 1 }, "reporter": { "localAgentHostPort": "jaeger-agent.default.svc.cluster.local:6831" } }

    To apply the config simply run:

    $  kubectl apply –f nginx-config.yaml

    Deploying the Sample Applications

    For simplicity, we’re using NGINX instances as our backend applications. We have two different applications, with App1 configured to redirect all HTTP traffic to App2.

    Both applications make use of ConfigMaps and volumes to configure the NGINX instances running inside the pods. One of the ConfigMaps has the NGINX configuration. The other is used to specify the tracer configuration for each NGINX instance (we include the configuration in each app, because as previously mentioned the app pods need to have access to the tracer to send information about the requests).

    Deploying App1

    Here’s the YAML manifest for the first application (app1.yaml):

    apiVersion: v1 kind: ConfigMap metadata: name: app1-config data: nginx.conf: |- user nginx; worker_processes 1; load_module modules/ngx_http_opentracing_module.so; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } http { opentracing_load_tracer /usr/local/lib/libjaegertracing_plugin.so /etc/jaeger-config.json; opentracing on; server { listen 80; server_name example.com; location / { opentracing_propagate_context; proxy_set_header Host $host; proxy_pass http://app2-svc:80; } } } --- apiVersion: v1 kind: ConfigMap metadata: name: jaeger-config-app1 data: jaeger-config.json: |- { "service_name": "app1", "sampler": { "type": "const", "param": 1 }, "reporter": { "localAgentHostPort": "jaeger-agent.default.svc.cluster.local:6831" } } --- apiVersion: apps/v1 kind: Deployment metadata: name: app1 spec: replicas: 1 selector: matchLabels: app: app1 template: metadata: labels: app: app1 spec: containers: - name: app1 image: opentracing/nginx-opentracing ports: - containerPort: 80 volumeMounts: - name: config-app1 mountPath: /etc/nginx/nginx.conf subPath: nginx.conf readOnly: true - name: config-jaeger mountPath: /etc/jaeger-config.json subPath: jaeger-config.json readOnly: true volumes: - name: config-app1 configMap: name: app1-config - name: config-jaeger configMap: name: jaeger-config-app1 --- apiVersion: v1 kind: Service metadata: name: app1-svc spec: ports: - port: 80 targetPort: 80 selector: app: app1

    To deploy the first application, run this command:

    $ kubectl apply –f app1.yaml

    Deploying App2

    Here’s the YAML manifest for the second application (app2.yaml):

    apiVersion: v1 kind: ConfigMap metadata: name: app2-config data: nginx.conf: |- user nginx; worker_processes 1; load_module modules/ngx_http_opentracing_module.so; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } http { opentracing_load_tracer /usr/local/lib/libjaegertracing_plugin.so /etc/jaeger-config.json; opentracing on; server { listen 80; server_name example.com; location / { opentracing_propagate_context; opentracing_tag app app2; return 200 "Success!\n"; } } } --- apiVersion: v1 kind: ConfigMap metadata: name: jaeger-config-app2 data: jaeger-config.json: |- { "service_name": "app2", "sampler": { "type": "const", "param": 1 }, "reporter": { "localAgentHostPort": "jaeger-agent.default.svc.cluster.local:6831" } } --- apiVersion: apps/v1 kind: Deployment metadata: name: app2 spec: replicas: 1 selector: matchLabels: app: app2 template: metadata: labels: app: app2 spec: containers: - name: app2 image: opentracing/nginx-opentracing ports: - containerPort: 80 volumeMounts: - name: config-app2 mountPath: /etc/nginx/nginx.conf subPath: nginx.conf readOnly: true - name: config-jaeger mountPath: /etc/jaeger-config.json subPath: jaeger-config.json readOnly: true volumes: - name: config-app2 configMap: name: app2-config - name: config-jaeger configMap: name: jaeger-config-app2 --- apiVersion: v1 kind: Service metadata: name: app2-svc spec: ports: - port: 80 targetPort: 80 selector: app: app2

    To deploy the second application, run this command:

    $ kubectl apply -f app2.yaml

    Deploying the Ingress Resource

    To enable access to App1 from outside the cluster, we create the following Ingress resource (opentracing-ingress.yaml):

    apiVersion: extensions/v1beta1 kind: Ingress metadata: name: opentracing-ingress annotations: nginx.org/location-snippets: | opentracing_propagate_context; spec: rules: - host: example.com http: paths: - path: / backend: serviceName: app1-svc servicePort: 80

    Note how we have used the nginx.org/location-snippets annotation to enable tracing context propagation at the Ingress Controller level. Snippet annotations are a way to add custom code to the final NGINX configuration in the Ingress Controller.

    Finally, we apply the Ingress resource by running this command:

    $ kubectl apply -f opentracing-ingress.yaml

    We verify that both App1 and App2 are running:

    $ kubectl get pods NAME READY STATUS RESTARTS AGE app1-68fd9db45c-szqpr 1/1 Running 0 53m app2-67c7788789-lvbgw 1/1 Running 0 53m

    Tracing a Request

    Now, we simply make a request to App1 through the Ingress Controller, where

    • IC_HTTP_PORT is the HTTP port in the Ingress Controller pod (80 by default)
    • IC_IP_ADDRESS is the IP address of the Ingress Controller pod. If you are using minikube, this is the minikube IP address.
    $ curl --resolve example.com:IC_HTTP_PORT:IC_IP_ADDRESS http://example.com:IC_HTTP_PORT/ --insecure Success!

    Reviewing the Trace

    We run the following command to enable access to the Jaeger UI, where JAEGER_POD is the “type/name” value for the pod we created from the all-in-one template in Deploying the Jaeger Tracer:

    $ kubectl port-forward JAEGER_POD 16686:16686

    Every request creates a new trace. To view the trace for the request we just made, we open the Jaeger UI at http://localhost:16686 in our browser, type nginx-egress in the Service field in the left column, and click the Find Traces button at the bottom of the column.

    Click nginx-ingress in the right column of the search window to select the trace. In the window that opens (below), we see the three spans for our request: the brown represents the Ingress controller, the blue is App1, and the yellow is App2.

    Click on a span to display more details about it, including the HTTP status code, hostname, IP address and Jaeger version.

    Conclusion

    Enabling OpenTracing for your services in Kubernetes helps you and your team understand what is happening in your applications and debug problems more quickly.

    Additionally, being able to trace requests starting at the Ingress Controller pod gives you complete insight into requests sent from outside the cluster as they pass through every service.

    Discover how the enhanced features in NGINX Plus can boost your Kubernetes deployment – start your free 30-day trial today or contact us to discuss your use cases.

    The post Enabling OpenTracing with the NGINX Ingress Controller for Kubernetes appeared first on NGINX.

    Posted by Web Monkey