13. Telemetry - Kiali Deeper Dive

So welcome back.

And in the previous video, we just set things up.

I hope either you've got a working system up and running, or you're happy just to sit back and watch these videos.

I said something at the end of the last video, which I think is so significant.

And I'm going to repeat it just in case you missed the end of the last video, I want to talk about exactly what do you need to have in place on your Kubernetes cluster? In order to get all of the upcoming benefits of Istio telemetry? Well, it's possibly simpler than you might think.

First of all, we do need to have those envoy sidecar proxies running in every pod that we're wanting to monitor.

Well, we've done that, let's confirm, keep CTL get po we know we've got two containers inside every pod.

And the second of those containers is that sidecar proxy.

Remember, you don't have to have every namespace in your system labelled with sidecar injection, it's up to you, which parts of the system that you have these proxies deployed to, for us, we're going to monitor the whole of this default namespace.

So we've achieved that requirement.

The other requirement to have telemetry working is that we do need the control plane up and running, what I'm leading to here, and I think it's so important, we don't need any specific istio yaml configuration.

If you read user guides, and blog posts, and so on, they'll immediately start going on about some fairly complex Istio concepts of virtual services, gateways, destination rules, and quite few other things.

Now, when I started with this, the other project that I was working with just wanted telemetry, actually, they just wants to do distributed tracing at first, and then they started using kiali, quite extensively.

And their big question for me was, well, we really don't want to set up an Istio gateway, that would have been quite a complex thing for them to install on their already existing architecture.

And they thought that this meant they couldn't use telemetry, well, I'm going to put a big red box around this, really just to emphasise that we don't need any clever Istio configuration.

We do need virtual services and gateways when we start doing traffic management.

And we will be covering that in detail on that section of the course.

But I've been quite deliberate in this section just to say you know what we're good to go.

We don't have to do anything particularly clever with our existing architecture, we can just go straight in and start using the user interfaces.

Now there is one big requirement that we need to make tracing work.

So we want to talk about that just now that's coming up in a few videos time, I'm afraid you are going to have to do some fairly serious changes to your application if you want the full benefits of tracing.

But this video is all about kiali.

So let's get started with that.

So what is kiali I'm sure most of you will have gone through the warm up section where we used kiali.

One of the great things about kiali is it doesn't take much learning really.

Now I've set things up so that kiali is available on port 31,000.

That's not actually standard.

If I do a kubectl get SVC dash n Istio system that will confirm that there is a kiali service here.

And I did just did an edit in the yaml file to change it to a node port.

And I just randomly chose Port 31,000.

Again, if you do the installing Istio video, then I'll show you exactly how to do that.

But it's a very simple operation.

There are other ways of getting there.

But I find that that's the easiest.

And the difficulty for me in this video is showing you everything I'm not showing saying when I think things are difficult to use or complicated or badly implemented.

But I can be really honest with kiali and say I think kiali is fantastic.

The perfect balance really have been incredibly powerful, it really can help you find very difficult to diagnose problems.

I tried to demonstrate that in the warm up session.

But also at the same time, it's pretty simple to use, it kind of makes it a bit hard for me to do a video on it because you know what, you could probably just play with it for an hour and and learn how to use it yourself.

So if I'm over gushing about how wonderful kiali is, I don't care.

It's one of the most valuable tools that I have.

So starting from the front dashboard, this gives you an overview of the overall health of the system.

And it splits this up by namespace so of course we only have two namespaces here but you can see very easily at a glance if your systems healthy.

Now, if you're seeing unhealthy at this stage, don't panic, it might just be the fact that you've come to here quite quickly, and we've Just started this system up.

So just while the system was warming up and the microservices were starting to talk to each other, there might have been some HTTP errors during that startup phase that being picked up.

Now a crucial aspect of kiali throughout, it's a very dynamic thing, it is not just showing you the health at a particular snapshot.

The top right is so important.

This is telling me this is showing the health over the last minute and it will refresh every 15 seconds.

So if I switch this, for example, back to the last 30 minutes, Yeah, perfect.

That's exactly what I wanted to see, that might look worrying.

We're getting warnings.

Now we're seeing that six of our while it calls them applications, these are actually pods, it's saying that four out of the six pods are showing some kind of a an ill health.

And as I say, That's absolutely nothing to worry about that was just in the startup period.

And if we wait 30 minutes, then this would all go to green.

So you're going to be a bit tactical in what you set for this period.

If for example, you've just done a request into the system, like I've just selected a vehicle there, and you know that doing that has caused the problem to happen.

And you want to be absolutely careful that your time frame here will actually capture that event.

If you set things to just the last minute.

Well, by the time you've started drilling down into here, you'll find that the fault is cleared.

So watch out for that, I must admit that has caught me out a few times that probably the first port of call has to be the graph.

If I click the graph link here, if you're not seeing anything on here, you might need to go to the namespace here.

If no namespace is selected, then you will get this warning here, but it tells you go to the namespace selector and select the namespace you want to see.

And I think the default graph you'll get is the version app graph.

I'll come on to that one in a minute.

But I think an easy one to understand is perhaps the service graph here.

And what that is showing you is with these triangles is every service that you have, in your standard Kubernetes yaml.

There's no special Istio stuff going on here.

And it's showing how they're connected together based on the traffic that has gone between these services in the period you can see here.

Do you remember a few moments ago, when I clicked I'm not going to click again.

But a few moments ago, I clicked on one of these vehicles here.

Well, I can tell you again, you don't need to know the inner workings of the system.

But that is implemented by a call from this service here, the Fleetman API gateway.

And it goes straight to this service here called the Fleetman staff service.

Now, I did that more than a minute ago.

And that's the reason that this line is being painted in grey.

I don't know how well you can see that on the video, I can zoom in either with this button here, or I can use my mouse wheel.

This line here is grey.

And that means in this time period in the last minute, there's been no traffic between these those two services.

Although there was some traffic a bit earlier than that.

Now what it's going to do is in a future refresh, eventually it will expire.

It will take that line out.

So be careful of that.

Again, I reckon if I go back across last 10 minutes, can you see now that line is in green, because in the last 10 minutes, there was traffic between that this service and that service.

Now that's useful, because we can see how dynamic This is.

If I go back to the last minute, we've got that line in grey.

But now if I switch to the system, I'm going to just click on a random vehicle and then go straight back to kiali.

Now I'm waiting for the next refresh.

Don't expect things that's great, there was a refresh there.

And you can tell by that and actually the line is still in grey.

So don't expect changes to be picked up immediately does take time for the telemetry data to propagate.

So I'm filling for time perfect.

That's the next update the next 15 second update and can you see now the line has been filled in in green to say there has been live traffic so it will take some time but if I were to carry on talking now for like 10 minutes and do nothing else, assuming no more traffic goes between that service and that service, this line will go to grey in the once this one minute window is expired.

I'll do a quick video editor to take us past that one minute window.

Yeah, that's about 415 second refreshes the line has gone to grey But I think it's important to note that if we wait a little longer, and I'm going to stop recording now and just leave this running in the background, eventually that line will be removed entirely if there's been no traffic between those two services.

And yes, I can confirm, I stopped recording there and waited for five minutes.

And once the five minutes have passed, then kiali determines that the traffic between those two services is now stale.

Remember, we had a call from API gateway to the staff service.

Now that's been removed from the diagram entirely.

So that's very important to note about kiali, then it's not showing you a static picture of your system architecture, it's only showing parts of the system that are sending and receiving traffic in the window that you've specified here, with an extra five minutes grace.

So if you ever find yourself looking at this and thinking, Well, my services disappeared, then you might need to either exercise the system by sending in a request or changing the window.

So let's look at that, then.

I reckon if I go back across the last 10 minutes, yeah, I can now see the full line from the gateway to the staff service.

If we go back to the last one minute, then that line is not in place.

But I can very easily Now again, I'll switch to the app, I'll click on one of the vehicles that should have sent a request from API gateway to staff service in the next refresh.

There was one refresh.

Now, that didn't change things.

Again, it takes a while.

Let's wait for the next refresh.

There's a refresh icon appears here when it's refreshing.

Interestingly, that refresh didn't catch it either, within about 30 seconds, since I made that request.

A bit nervous now.

There's an extra refresh.

And yeah, actually, by the time it's actually captured that data, already, that line has gone back to a grey line set actually took a little over a minute for that data to actually appear on the graph.

So be aware of that.

You might have noticed in these demos that the layouts have been changing a little bit remote, you can't expect it to always build a beautiful, perfectly drawn graph.

Sometimes these things can look a little bit messy, you do have some options.

Here, you have three alternative layouts, the default is the one here.

But if I click on the other one, I'm not sure how to describe these.

And I'm certainly not familiar with the implementations of these layouts.

But I always find layout number one's a bit more angular, it's a bit more sort of straight edge.

And layout.

Number two, well, it just looks different.

I don't really know how to explain that.

Just experiment for yourself with three layouts and work with the one you prefer.

I think on balance, I mean, the the default layout tries to lay things out in a kind of left to right fashion.

Where is it more of a tree style.

Whereas layouts one and two, I think are a bit more of a star formation.

I think oftentimes, I prefer personally layout one, but that's entirely up to you.

So that's the service graph.

And it's probably the graph where I spend most of my time in kiali.

Before I leave the graph, I should mention that, obviously, in real life, some of these graphs can get very big and complicated.

Although it's not usually that bad, because you can always filter by the particular net by namespaces.

namespaces shouldn't really have too many items in them.

Although I realise in the real world, lots of projects do go stuffing hundreds of things into namespaces.

So one useful tip might be if you have a particularly complicated graph, you can always take any of these items, I'll go for this one here doesn't matter what it is, if you double click on any item, then the view will change to show you just that item together with any incoming and outgoing routes from that particular object.

So that can be really useful to clean things up.

And you can always go back to the full graph here.

Now as a service graph, so each of these triangles is representing a regular Kubernetes service.

But we have some alternative graphs here on the drop down, and there are four of them.

Now until we go a little further in the course I can't explain all of them, we will need to do some traffic management before I can give you a full description of the version dap graph, but I can show you the workload graph, and you'll immediately notice that it is a little bit bigger.

Looks a little bit more complicated at first view.

Let's just try the different layouts and see if we get something a bit a bit cleaner.

I think that's all Probably the best one.

Now what we have here is a view where we still have the services.

So these are the triangles here.

And by the way, there's a button here to show you the legend or the key for the graphs of the triangles of the services.

But on this graph, we also have circles, representing what they call the workloads, you can think of those has just been the parts.

This diagram is perhaps not that useful with my particular system, because really, every service just has a pod behind it.

I should mention if you go further with this, so if you have multiple pods for a service, you will only still see one pod, you will only see one workload, I should say.

That's because the workload represents all of the pods for a particular service, I don't find this graph quite so useful, because it's a lot more cluttered, because you always have basically two entries for everything, you have the service, and then its underlying collection of pods, if you like.

But there's at least one extra thing you can do with this graph that you can't do with the other.

If I go to this button here for edge labels, there's a very useful option of setting the response time.

Now the response time is telling you the average response time for a particular set of requests.

So I can see, for example, when the position tracker is servicing a request, its average response time is 237 milliseconds.

Now, you'll notice that this label is only in position between the service and its underlying workload or its underlying parts.

So for that reason, if you switch to the service graph, even with response time selected, you don't see anything.

So don't forget that if you do want to see these response times, then you will need to make sure you're on the workload graph.

Looking at the clock, I think I will defer talking about the version that graph on the app graph.

Until the traffic management section of the course that's really where those graphs come into their own.

For now, at least, you won't see a big difference between those graphs and the workload graph that we've just seen.

So what to do next well, on any of these graphs, you can right click on any of the objects.

So I'm back here on the service graph.

And if I right click on the any of them Fleetman staff service, we get a pop up menu, and we can go into the details of the service.

This will very usefully show me a list of all of the pods that are represented by this particular service.

And I also have an inbound metrics graph here, which will show me things such as the request volume, the request duration, over time, the request size and the response size.

So all of that might be very useful.

If you are interested in these metrics, you're probably more likely to be looking in grafana, which I'll be showing you in a short while.

So I won't go into too much detail there.

By contrast, if I go back to the graph, that was the service graph, if I'm on the workload graph, there is a difference.

Remember, right clicking on a service will allow me to look at details traffic and inbound metrics.

But if I right click on a workload, then I get different options.

And I can access all of these options.

Actually, through the Show Details link, very usefully for a workload, you can look at the logs.

Now remember, a workload can be more than one part.

So you'll find a drop down list here of all of the pods for that particular workload.

We only have one pod, per workload in this system.

But remember, of course, all of our pods have two containers in them the application container and the proxy.

So you can look at the logs for the proxy as well.

Now, of course, you can do this on the command line.

There's nothing particularly special about this.

But I have found that useful a few times where I'm debugging an emergency situation.

And I can stay within this user interface rather than having switched to the command line.

So maybe not a big deal for you.

But definitely nice to know it's there.

Notice that the logs work differently to the rest of kiali.

It has its own time period, and you have to manually refresh them.

So keep that in mind.

Now, I don't know how if you're following along I don't know how chatty this particular part is.

So you might not see anything.

I'm seeing lots of exceptions in there, which I think is there purely because I've just restarted we still need to revisit the version That graph, which we will do in the traffic management system, just a few miscellaneous things really, you might find useful under the display button here, you can click on traffic animation.

And that will give you a cue animation showing you the direction of the traffic flow, I switch to the service graph, and you'll see it works there.

Just the same, this perhaps is a little bit of eye candy, because you can see the direction from the arrows.

But quite often, I do like to have this on, it just gives me a stronger visual feel for the direction that things are happening.

This icon that's moving down the line also changes depending on the amount of traffic going along that connection.

And also, it will make the little circles bigger, depending on the size of the payloads between them.

We're not really getting a feel for that on this demo, but on your systems that might be really useful, are bound to have missed quite a few things.

But I hope you will find this quite intuitive to work with.

And if you do run into any problems, then the documentation, which you can get here by following the icon here on the top right to documentation, I found that documentation to be excellently well written.

One last thing I would like to cover.

And this is something that we really do need to have done the traffic management system to get a full feel for but it's just too big really to leave until later.

So I'm going to do it now.

And that is you might be amazed to discover that as well as just looking at the system as well as just observing what's going on.

You can also make dramatic changes to the configuration of this system.

You can actually make modifications to your system architecture through kiali.

Well, that's a big feature.

So I think that definitely needs a video of its own.

We'll do that next So welcome back.