17. Telemetry - Why you need to "Propagate Headers"

At the beginning of this course, I set out that the design of the course is that you're working on a regular Kubernetes system with no special features added to it.

And I'm demonstrating that Istio can be used as a service mesh.

And we get lots of wonderful features such as telemetry in this section, traffic management coming next, and then policies and security.

And none of that needs you to alter your code, the Kubernetes pods that we're deploying remain on altered.

So Istio aims to be a non invasive service mesh, it doesn't affect your code.

But I did warn you, there is one exception.

And that is, if you want distributed tracing to work, then you do need to make alterations to your code.

And now I can finally confess that in the previous videos where I was demonstrating Jaeger, I had made changes to the running code inside my pods, so that everything worked.

So what have I done? And what do you need to do in your systems, if you want distributed tracing, like yay, go to work, the high level explanation really is that as we have a chain of requests running through your architecture, somehow the distributed tracing mechanism needs to work out that all of these requests are for a single trace.

In other words, it needs to correlate these requests.

The way it does, this is semi automatic, I've gone back to Jaeger here, and I've just picked a particular span from a trace.

And I showed you that you can drill down into these spans.

And you can look into the tags, for example, we can look at the URL and the method that's being called.

But one field that I didn't point out was in every one of these spans, there is a field called x request ID.

And it's good.

It's just a randomly generated, globally unique identifier.

And what you'll find is, if we just make a note of this request ID, it begins 954 and ns three to a, if I pick another one of the spans, such as this one here, which is quite far away from the original one, if I drill down into that one, can you see if you remember, I think this is right, that request ID 954, ending in three to eight is the same x request ID.

And you'll find that all of these spans in this trace have the same request ID.

And that's the mechanism that's going on under the hood here.

If requests have the same request ID, then they're all grouped together into a span.

But there's a big problem.

The problem is where do these request IDs come from, you might be thinking that you've got to go into your code and somehow put these request IDs into your requests actually don't.

These are automatically generated by Istio, you don't have to worry about them.

And then back to this scenario, if I wind everything back, what will happen is when a request comes in to your cluster, such as into the web app, and then the web app makes an outgoing request, in this case, it's making a request to the API gateway.

Well, you know, that those requests are going to be routed are routed through the envoy proxy.

One of the pieces of logic in this proxy is if the incoming request did not contain an X request ID, and it won't, because we've just made in a request from a browser.

Then, when the proxy is calling the target Potts proxy, it will automatically add in a header x request ID and it will automatically generate that gu ID.

I couldn't be bothered to put the gu IDs on here.

So I'm just using a random number.

But the principle still applies.

Here's the big catch.

What will happen next is the target proxy is going to invoke the local containers logic.

So this container will do whatever it needs to do.

Then this container decides it wants to call this staff service container.

So the process continues.

There's going to be an outgoing call, routed or routed through the proxy.

Now you might be expecting well that's great when the ex request ID came into here.

So perhaps automatic It can be sent outwards.

Well, unfortunately, not.

Don't take my word for it.

I'm here in the documentation for Istio under the distributed tracing questions.

And the second question here is what is required for distributed tracing with Istio.

If I follow that link, it says here in order for various trace spans to be stitched together for a complete view, in other words, for a complete trace applications, and that means your code must propagate.

Well, they call it here, the trace context, they really mean that x request ID header.

Now, it's a bit more than that.

It's the x request ID, and also a set of headers beginning x dash B three.

So that means when, for example, this container here made its outgoing request, remember, your containers code will have visibility of this incoming request ID.

But your container has to put that request ID into the outgoing request.

If you don't do that, then what will happen is, when this outgoing request is made, this original request ID is lost.

And that logic is going to kick in again, this proxy here is going to send the request of the target pod.

The logic in this proxy will be Oh, I didn't receive a request ID here, it's this line here that I'm pointing to with my mouse.

If that header isn't in this line, then the proxy here will not receive the header.

So we'll run that logic again, it will say, oh, okay, I don't have an x request, ID or better generate a new one.

And it will be a completely new gu ID that is generated.

So the end result of all of this, if you don't do that work yourself, you'll end up with several traces, each of which are just containing one or two spans.

So it will all be kind of split up, which is not what you want.

So to run through that, again, if you want distributed tracing to work properly, in other words, for the traces to contain all of the spans.

And this is what you're going to have to do.

So we have our initial request here, which goes to the first container.

This is this containers making the request, which is routed or routed through the proxy, the proxy logic is intelligent enough to know there's no request ID on here.

So I need to generate one that will be a big ugly gu ID.

And that big, ugly, gooey sense of the target proxy, a target proxy will invoke the logic in this container.

And I want to be clear here that this x request ID will have been sent into this container.

But then when you make an outgoing request from this container, this line here, you are highlighting this in red here, you have to send that value that you received, and you have to send it in your outgoing request.

So this time around, because the proxies received this x request ID, it will know this time, it doesn't need to generate a new one, it will send it onwards to the next proxy in the chain.

And the fact now that we've got identical request IDs will mean that these spans can be gathered together, I'm really sorry to have to tell you that because this is invasive, it means you have to make changes to your application code.

If you don't do this, then you know the world isn't going to end.

But it will mean that your traces are not going to stitch together.

And the really disappointing news is, of course, really, you have to do this in every single container that you're going to be deploying into your architecture, again, the world world.

And if you forget to do this in one container, it's just that any traffic flowing through there is going to cause breaks in the traces and don't take my word for it.

I'm here on the frequently asked questions for Istio.

And one of the frequently asked questions on the distributed tracing is certainly a big frequently asked question for me.

And that is, why can't istio do this for me? Why can't it propagate the headers instead of the application? Now they don't really go into any detail in the answer other than saying that the Istio sidecar.

In other words, the envoy proxy has no implicit way of correlating the outbound requests to the inbound requests that cause them.

So if you think about it probably makes sense this request coming in here with id 47.

If then this container does some processing and sends an outgoing request, in the absence of that x request ID field, this proxy here has no way of knowing that this request was caused by this incoming request.

And I have to admit that Yeah, although I'm disappointed, that does appear that this isn't really possible.

Now, I wouldn't be surprised if in a future version of Istio, somebody comes up with a way of solving this problem.

And if you have a solution for this problem, then contact the Istio developers because I'm sure they'd love to hear from you.

I think this is definitely the weakest area of Istio.

But it does look like a bit of an unsolvable problem, given the architecture of these proxies.

So for whatever reason, it's certainly the case at the time of recording in 2019, that you do have to manually inside the container logic here.

When you receive a request, you've got to somehow store this value in some variable somewhere or a database.

And then be sure that when you make an outgoing request, you add this header to the new request that you're sending out, got to do it manually.

And we'll keep an eye on this.

And if anytime in the future in a future version of Istio, this no longer becomes a requirement, I will notify everybody and rerecord this video.

But for now, it's necessary.

There are a million different languages out there.

And I don't know what programming languages you're using.

But clearly, every single language will have a different way of enabling you to forward these headers.

I'm hearing the reference manual of Istio.

And if I look under the tasks link, and then follow the link to telemetry and distributed tracing, I think we'll look in the overview.

Yeah.

And in the overview, the first thing they talk about is the trace context propagation.

That's exactly what we're talking about here.

istio needs hints to tie together the entire trace, our applications need to propagate the HTTP headers, they give you a list there of the headers that we need to propagate.

And they're showing here, a code section from their sample application, which happens to be written in Python.

I don't know if you're familiar with Python, I don't use Python very often.

But they've given a block of code here where they're extracting these headers.

Going a little further down, there is a part of their sample application that uses Java.

They're using some technique here, to Yeah, emanate, this is really ugly, because basically, you've got to do this in every single one of your methods inside methods or operations or functions inside your applications.

So what they're doing here is presumably, this is a method that is implementing a REST endpoint.

So this will receive GET requests.

And on receipt of a get request, Java calls this an annotation and the annotation is saying, I want the web framework, the rest framework that we're using here to automatically grab this line here is the important one, grab whatever is contained in the header, x request ID, and store it in a local variable called x rec, then they've gone on to do the other tracing fields.

And then I don't know what they're doing here.

Exactly.

But presumably, this is calling an upstream service.

And they're being careful to send as a parameter, all of those headers, if absolutely horrible, so it isn't pleasant.

And I can't give specific advice for your programming languages.

But if you're using spring boots, and I'm only saying this because the Fleetman application is implemented in spring boots, now I'm here in the GitHub repository for for this course, you'll find this@github.com forward slash dick chest the word forward slash istio dash Fleetman.

So I've got the source code for the Fleetman in here.

Now, if you go to the releases link here, I'm generating a series of Docker images, one for each section of the course.

And the Docker images you've been working with in the previous videos were tagged colon six, and if you look down the releases list, you should find a release, called telemetry.

Now for that release, I've created a GitHub tag.

And I just make the GitHub tags be the same as the Docker image tags.

If you follow this link here to tag sex, I'm doing that just so you get specifically, the source code.

This isn't the master, this isn't the head of the repository.

This is the source code that was used for the system that you've just been monitoring in the previous videos.

And I just want to show you that what I've done to achieve this propagation is a Spring Boot trick.

Now when I came to implement this, I was very lucky in that in all of these microservices, I'm using a library called Fein.

I don't know if you're familiar with fame.

But if you are, it's a library that just enables you to do rest calls.

Without having to make a network request yourself.

It gives you a very high level way of calling URLs, basically.

And I was really lucky that in using Fein, and I didn't use fame for this reason, I just like fame.

But one of the features of Fein is it can do automatic header propagation.

And it means that if I take any one of my microservices, so looking here, the Istio Fleetman API gateway, if I drill down into the source folder, and then the main folder, and then into Java comm, virtual pair programmers, and then into API, we love our folder structures in Java, I'm just looking for anywhere, we're really where I'm making an outgoing request.

So think here in services, I have position tracking external service dot java.

And let's go for, for example, line 27.

Here, this is where I'm calling get history for a vehicle.

And that is going to be an external service.

You'll notice noticed in the previous demos that you saw this call where we get that history track.

And the way that works in fame is we create a file here I've called it remotes position micro service calls dot java.

And it's really nice, we can just write this as if it were in regular Java function, a regular Java method, get history far, this all looks very Java ish.

And just by this annotation on here, Fane will automatically convert that into a network request.

It's very neat.

It's very nice.

But what I'm trying to show you here is, I don't if you noticed, there was no header stuff there at all.

I never mentioned that x request ID.

And the reason for that is and I don't have to, I've just gone up a few levels to find this config folder.

And inside there, there is one Java class called propagate headers, interceptor dot java.

I've put a great big comments on this class, because I know many of you might want to use this in your own projects.

But if I can just run through my comment here, this is what's called an interceptor in fain, this is going to run automatically, I don't have to just have to drop this into the project, and it will automatically fire up whenever there's an incoming REST request.

If there's an incoming request rest requests, we will take all the headers beginning x, and then we'll store them away.

And we'll ask fain to add them to the corresponding outgoing rest requests, provided we do this in every single micro service, then that's going to have the end result of propagating the custom headers that we need for distributed tracing.

The class isn't particularly pleasant, but the great thing is, this is all that's needed.

I don't have to call this class, it will be automatically called by the framework, which is superb.

So what I would do in real life is I would put this class in some kind of library, and then use that library across all of my micro services.

For this system, I've just kept things simple, and I've just made sure that every single one of them has this file in it somewhere.

I've just copied and pasted To be honest, but it works for this demo.

So if I switch across, for example, to the staff service, and going to source main Java There's a config folder again.

And again we have the propagate headers interceptor dot java exactly the same class.

It's never called, but the framework uses it to propagate the headers.

I realised for many of you, that won't make sense if you don't work in Spring Boot, and if you're not working with fain, that's as far as I can go in just saying, whatever programming language you're in, try to have a look around and see if there's a library that will do this for you.