Stefan (00:03):

Hello and welcome to Automate It, from the Automated Global World Capital Studios on the 75th floor of Polymath Headquarters in the San Francisco Mission District. I, of course, am Stefan Seltz-Axmacher. And with me is, I don't know, some sort of a guy wearing an R2-D2 shirt for some reason. Yesterday was Rosinante. Well, who exactly are you again?

Ilia (00:24):

Big fan of sci-fi and robotics in general.

Stefan (00:29):

Of course, this is Ilia Baranov with me today and you've tuned into Automate It. Every week we talk about how to make robots and how to not make robots. We like to start every episode off with my favorite game, where we pick a random technology and a random setting for us to try to spitball a robot business out of. Sometimes being very weird, sometimes being more or less something that you could see being an actual company. So for this week Ilia, what's our broader topic going to be after this initial little game?

Ilia (01:00):

So as we hinted last episode, we're going to be talking about how to measure robots in general.

Stefan (01:05):

That can't be that hard. That will be a doozy, we can spend more time on this initial game. Well, what's our setting this week, Ilia?

Ilia (01:11):

All right. Let's see, we got office.

Stefan (01:14):

Office. And for the office robots, we will be using...

Ilia (01:21):

You're just rejecting a whole bunch of them.

Stefan (01:23):

Yeah, we're rejecting a whole bunch of them but actually this one-

Ilia (01:27):

I feel some bias.

Stefan (01:28):

Yeah, there's definitely some bias in my selection. We'll be using VR for an office robot.

Ilia (01:34):

Oh, nice. I have some firsthand experience.

Stefan (01:37):

It's why I ran through this. And I'd say this is going to be one where we could stumble into some real companies that we don't want to do, and that's both from an interesting podcast and from a business perspective. I guess, what are you talking about in terms of your experience of office VR robots?

Ilia (01:54):

Yeah, let me start off with a little bit of a story. When Clearpath Robotics took over the PR2 program, I noticed that, this is right around the time the very first Oculus had come out and that kind of VR was just taking off. And I noticed somebody had put together a package to output 3D data from ARvis in ROS; which is the visualization data tool inside of ROS; to this VR headset, specifically for the PR2. And so as I was trying to learn the PR2, I set myself a goal to learn how to use this package to try to figure out what was going on. And you could move your head with the VR headset and the robot's head would move as well.

Stefan (01:54):

Mm-hmm.

Ilia (02:29):

And it would-

Stefan (02:30):

Like an Apache attack helicopter.

Ilia (02:31):

Yeah. Much less threatening and much slower, but yeah, same idea. And the data from the LiDars and the depth camera and all that would be projected in this virtual reality augmented view where you could kind of see beyond the borders of the sensors. You could see the physical position of the arms and that kind of stuff. Anyway, the fun part of the story was Clearpath decided to open their new offices with a ribbon cutting and they wanted me to tele-operate the robot to come and cut the ribbon. And-

Stefan (03:00):

So you had a robot driving around with scissors at high speeds around crowded room?

Ilia (03:05):

Yeah. Maybe not driving, maybe just standing still but waving its arms around at least.

Stefan (03:09):

That's not a robot I want to be near.

Ilia (03:12):

Yeah, there was a nice safety barrier.

Stefan (03:14):

Yeah and Kevlar vests worn by all.

Ilia (03:17):

The funny part was I didn't want to just control the arms using a joystick or something. I wanted to use a Microsoft Connect to capture my skeleton and reproduce it on the robot. So if I moved, the robot would move and copy me. The problem was-

Stefan (03:17):

A bit like some of the Gundam.

Ilia (03:17):

Yeah.

Stefan (03:30):

Some of the Gundam controls.

Ilia (03:32):

Exactly. So ultimate nerd dream. The problem ended up being that the connect output of the skeleton position was really noisy, so the robot looked like it had Parkinson's. Anytime it would move, it would shake crazy. You could never get it. So we couldn't get it accurate enough to even cut the tape. What we ended up doing, just any good ribbon cutting, is we precut a segment of the tape, taped it back on-

Stefan (04:04):

Karate chop?

Ilia (04:04):

And it just karate chopped it. And that's how we got the ribbon cutting.

Stefan (04:08):

So that being said, I think we obviously aren't going to use Connects to do body matching for our office robot. I think in terms of office robots, I don't want to be Double Robotics like that. Double Robotics, they've been around for a while. It's an interesting product. It's kind of the business that it is. They've accomplished a lot of cool stuff. I don't think we should talk about telepresence robots. That's been there, done that. I think that maybe, one of the areas that robots can help the office is in team building exercises more than telepresence.

Ilia (04:43):

With VR?

Stefan (04:44):

Yeah, with VR.

Ilia (04:45):

Okay.

Stefan (04:45):

Because if you think about it, in this world where people are increasingly remote, it seems like a lot of people are re-clustering back in office but there might be a couple of long-term stragglers. People who aren't going to live near the office, but they should still be able to be involved in the team building events. And there's usually a lot of stuff going on in team building. So you need a telepresence robot that can see 360 degrees so that the person in VR can really feel like they're a part of the team building exercise, whether it's a cheesy guy coming in and figuring out what your spirit animal is, or a happy hour down the street.

Ilia (05:26):

So actually really great note there by the way, the 360 camera part that you mentioned, you skim off of it but that's actually really key to making that work. You're trying to put a camera on a pan tilt.

Stefan (05:26):

Yeah.

Ilia (05:36):

The delay between your head motion and the camera head motion, it's going to give you horrendous nausea. Again, first hand experience.

Stefan (05:42):

So we need to see 360 degrees.

Ilia (05:45):

360 degrees is clever because-

Stefan (05:46):

You could just cover the whole robot with cameras.

Ilia (05:48):

Well maybe and synthesize the view.

Stefan (05:48):

Yeah.

Ilia (05:51):

But the idea is that when you move your VR headset, you're moving in a already pre-rendered image sphere, and so you won't get that motion sickness. Trying to do pan tilt, it just, ah, it's terrible.

Stefan (06:01):

Okay. So basically we need a bunch of 360 cameras on a large movable platform. And I think team building exercises include things that are off campus. So it needs to be able to work at least to the nearby bar, at least to the nearby lunch spot.

Ilia (06:24):

We'll get back to the lunch spot, but I was taking this the other way. When I'm thinking of team building, I'm thinking about trust falls. And so what you do is you have the robot try to catch a person and you don't know if the other person's just a jerk or if there's just a software bug it moved down way. And your team member can have full confidence of just being like, "Oh, you fell. Oh, it must be a bug. Sorry." Perfect.

Stefan (06:46):

What was the direction... Sorry. You're thinking team building trust fall robot?

Ilia (06:50):

Well, team building classic, the more summer camp team building style.

Stefan (06:55):

Yeah.

Ilia (06:56):

Not-

Stefan (06:56):

Or you can have the robot go canoeing?

Ilia (06:58):

Yeah, that'd be super effective.

Stefan (07:03):

Hopefully you put it in the middle. Maybe you can crouch a little bit along its wheelbase.

Ilia (07:08):

Oh boy. Is that Can inia activity, canoeing.

Stefan (07:14):

The problem that I have in the team building and it also reminds me too much of things people have actually tried to do. What's a weirder wackier in-office VR robot?

Ilia (07:24):

Yeah, so you want to change sides. Well, the classic, I mean the obvious one is Telebrones, but let's what's even stranger?

Stefan (07:30):

Yeah?

Ilia (07:31):

Boon.

Stefan (07:32):

You know what you could do, actually?

Ilia (07:36):

What?

Stefan (07:37):

So robots a really stupid platform to have much of cameras on because we all have a camera on our laptop. We might have an exterior webcam, whatever. The problem is sometimes you try to ping somebody and you're remote and you're not with them. So essentially you try to slack and you try to email them, you try to call them out, or maybe they walked away. What we need is a drone with a long poking stick, like in the episode of Bob's Burgers. Both the drones-

Ilia (08:03):

Your folder?

Stefan (08:04):

To get half of your shoulder, and we don't know where in the office they might be. So use the VR to be able more effectively, quickly pilot, do drone racing, flying around the office to find the people you're hunting for.

Ilia (08:16):

With a stick?

Stefan (08:16):

Yeah.

Ilia (08:17):

I feel like safety glasses are going to be mandatory at this office.

Stefan (08:20):

And that is the price we pay for a broader eligible workforce.

Ilia (08:26):

Yeah. I mean instead of cur-

Stefan (08:29):

Oh, and also we'll probably need these in afore tomes.

Ilia (08:32):

Yeah, exactly. Let's get-

Stefan (08:34):

People are from home.

Ilia (08:35):

Yeah.

Stefan (08:36):

We've all seen those Kick-starters for the companies that never shipped of the really capable drones that were about six inches by six inches.

Ilia (08:36):

Yeah, you could just do all the things.

Stefan (08:45):

Yeah, they can do all the things. So we basically put one of those at every employee's home office, and since it's work hours, if it's work hours, anyone on the team can take control of it, fly through their house, find them, and tap them on the shoulder.

Ilia (09:00):

Or in my case, when I bike to work and I'm on a call-

Stefan (09:02):

Well yeah-

Ilia (09:03):

It has to chase me down the bike lane.

Stefan (09:05):

There might be a bigger one. There might be one that's more like the size of a reaper drone. That flies in low behind you and then drops us swarm of these that then chases you down to tap you on the shoulder while you're riding your bike. Just so I can ask you, "Hey, by the way, where's our EIN number?"

Ilia (09:23):

Yeah, I mean, why would you ever just install an app on your phone, has a notification?

Stefan (09:27):

What if you have your phone on silent? I need to be able to proactively ping people even when they don't want to be pinged.

Ilia (09:33):

I mean, you showed me your phone with the 400 and something notifications. I've just decided if I ever need to get a hold of you, I have to call you. There's just more other-

Stefan (09:43):

That's why texting, that's not work.

Ilia (09:43):

Yeah.

Stefan (09:44):

So if we're building this universal poker, we basically want to, it needs a nesting pod that you can put on your desk, plug in, that it can easily fly to and from. It needs, I don't know-

Ilia (09:56):

That's a self charge, right? So...

Stefan (09:58):

Yeah. From that nesting pod.

Ilia (09:59):

Yeah. There's no complexity on drone self-charging.

Stefan (10:02):

Well it's just on that pod and then some copper exposed.

Ilia (10:06):

It's still wireless charging?

Stefan (10:07):

Yeah, inductive charge. It's all good.

Ilia (10:09):

Yeah, that's pod.

Stefan (10:10):

Probably, we might need Gen2 will be able to hunt you down during your commute, but Gen1 will just stay within your house to keep this problem simpler.

Ilia (10:21):

And VR tele-operated. So no real intelligence, right?

Stefan (10:24):

Yeah-

Ilia (10:24):

It's just VR?

Stefan (10:25):

Well, I mean once I get the information that I'm interested in, I do tend to change subjects rather quickly. So I might need it to be able to fly itself back to where it started and park itself.

Ilia (10:39):

Well, in classic novice and Robotics, the immediate thought would be like, oh, just replay all your commands backwards. That should be fine, right?

Stefan (10:47):

Yeah. Why can't you do that?

Ilia (10:48):

Noise. The real world has noise and disturbances.

Stefan (10:52):

But I mean if you're flying inside, there's not that much wind.

Ilia (10:54):

Oh yeah. There's downdraft from your own rotors.

Stefan (10:58):

Okay.

Ilia (11:00):

Even a ground robot with really accurate encoders. If you try to get it to replicate, play back the same motion over and over, it'll drift off path after two, three iterations. Really quick.

Stefan (11:10):

Tough out.

Ilia (11:10):

Yeah.

Stefan (11:11):

So we need to make it autonomous?

Ilia (11:14):

Yeah, we send you autonomous.

Stefan (11:15):

Yep.

Ilia (11:15):

Yeah.

Stefan (11:15):

We need to... And it could pre-map. This drone could regularly take off of your desk, start flying around your house, mapping out every-

Ilia (11:26):

Oh.

Stefan (11:26):

To see where everything is.

Ilia (11:27):

I think if it's been VR piloted.

Stefan (11:30):

Yeah.

Ilia (11:30):

At least, it's not going to exactly repeat the same commands to come back, but it already knows the reasonable path to take.

Stefan (11:35):

Yeah.

Ilia (11:36):

Right. So it could take the path it took to get there and play it back but just control to that path, not just dumbly repeat the same thing.

Stefan (11:36):

Yeah,

Ilia (11:43):

Yeah. So it doesn't even need a map at that point. It's just like...

Stefan (11:46):

Well, it would, but okay. Yeah.

Ilia (11:48):

Localize, just local featured.

Stefan (11:50):

Yeah.

Ilia (11:51):

Not trying to build a global path-

Stefan (11:52):

Within your house?

Ilia (11:53):

Yeah, exactly. Just optical flow, which is a software trick where from your camera lens, as you see things move by, if you see something move by to the right in your camera frame, you know your camera has moved to the left. So you have that optical flow software, you get roughly accurate, good enough to navigate a few dozen meters kind of thing.

Stefan (12:12):

And then is there going to be any complication? I imagine this thing will be on wifi. We'll require our employer employees have good wifi connections. Is there any challenge with us doing live video streaming off that little drone? Our wifi to my VR headset?

Ilia (12:28):

Yeah. We won't get very high quality. We won't get very large field of view or very high frame rate. But you could get something like, I don't know, I'm guessing 10 frames a second middling resolution.

Stefan (12:43):

So as long as we don't hire hoarders, we'll be relatively able to fly around?

Ilia (12:47):

Fly around, yeah.

Stefan (12:49):

Yeah.

Ilia (12:49):

A rough view of the world.

Stefan (12:50):

We'll say this is only a tool for people who aren't hoarders. We're also okay with incredible invasions into their home space.

Ilia (12:58):

Yeah. It's not a problem to have a drone piloted by random office-

Stefan (13:02):

Employees. Yeah.

Ilia (13:02):

Employees. Yeah. Potentially, office rivals of some kind.

Stefan (13:08):

Security will be a real issue with this particular drone. I think people might care about the security on this more than they care about actual predator terms. Well, how would it mess up our controls models if it's essentially a quad copter with a, call it 18 inch pole sticking out the side of it, or an 18 inch antenna used for poking.

Ilia (13:31):

How heavy is that antenna?

Stefan (13:33):

I mean, antenna. Wait...

Ilia (13:37):

Assuming-

Stefan (13:38):

Thinking car antenna level of thickness and with a nub on the end, so it's a little less pointy.

Ilia (13:47):

Few hundred grams?

Stefan (13:48):

Yeah.

Ilia (13:49):

I think that'll depend. Maybe if it's one of these little pocket drones, I think that's just going to be a no go. It's not going to lift off. Forget controls, it's just the motors won't be powerful. But if you're thinking of something like a DGI sized drone-

Stefan (14:00):

Well, the other week in our office, somebody took one of those off right next to conference room for reasons that are still-

Ilia (14:08):

Still unclear.

Stefan (14:08):

Still unclear. Just those people we never saw before in our office that has 10 people working it, fly the drone around. They were like, "What is going on here?" So that was really loud. I think that might hurt productivity. I think it needs to be a pocket drone. How long of a poker film you have on a pocket drone?

Ilia (14:24):

Nothing. Basically these pocket drones have maybe a payload of a few grams. They're really working right on the hairy edge of functional. Yeah, they're built to barely lift off kind of thing. So I think-

Stefan (14:38):

Maybe a ball would stick?

Ilia (14:41):

How about we get rid of the stick.

Stefan (14:42):

Yeah.

Ilia (14:43):

And all we do is that the pocket drone has a cage around it. So just a thin plastic cage-

Stefan (14:50):

Just hits you.

Ilia (14:50):

It just hits you. Exactly. It just bump, it physically bumps into you because you'd want the cage anyway so that the drone can be [inaudible 00:14:56] and it's not too bad.

Stefan (14:58):

So since we're, I think at the time of the segment anyway, coming to a terrible manager near you, the Kamikaze 7,000.

Ilia (15:07):

No-

Stefan (15:07):

Have you-

Ilia (15:09):

No. It's a terrible manager, right?

Stefan (15:10):

Yeah.

Ilia (15:11):

It's the ball-buster.

Stefan (15:16):

The ball buzzard buzzer 7,000. Now you're a terrible manager. Kennedy Infringe on your most private moments by flying a throne launched inside your house, directly into your crotch. Available to any company willing to pay the $400 per month subscription, minimum order size five robots through your annual commitment.

Ilia (15:38):

And inevitable lawsuits. All right, so moving on. How will we measure this terrible robot's success?

Stefan (15:49):

Oh, I mean, I think to put it a different way, to put it in a way that rhymes with reality much more. Bailey, I'm CEO of our little company, a Polymath Robotics.

Ilia (15:58):

No?

Stefan (15:59):

Yep.

Ilia (16:00):

Oh God.

Stefan (16:02):

And Ball-buster 7,000 to count other companies as well. I finally, I could ask a lot how we're doing. How's your progress going. Is the robot robotting yet? How well can it do the robot? Give me numbers. And I think it will inspire our team to type code faster. If there is a dashboard on the wall with numbers that just decrease and increase, and every time they see that number get bigger, they know our robot is more robot-y.

Ilia (16:35):

I love a good dashboard-

Stefan (16:36):

Yeah.

Ilia (16:36):

To be serious.

Stefan (16:37):

Yeah.

Stefan (16:37):

Good dashboard makes a big difference.

Stefan (16:39):

So hey, I mean there's a revenue dashboard, that one's easy.

Ilia (16:43):

Yeah.

Stefan (16:44):

Yeah. I guess outside of the business side, what metrics can we throw on this dashboard so I know that we're robotting better than any of the other robots?

Ilia (16:52):

Yeah, I mean fundamental ones if you're operating a fleet like we do, is number of robots.

Stefan (16:57):

Yeah. We can count that. That's good.

Ilia (16:58):

We can count that. Where are they in the world?

Stefan (17:01):

Yep. I guess what I really want to know is are the robots working?

Ilia (17:05):

Working?

Stefan (17:06):

Are the robots better than they were last week?

Ilia (17:10):

Yeah. So that's the thing, is one of the actual functional metrics we want to use is we want to start counting things like how far did we drive autonomously last week? Did we drive more this week? If we drove more, was it valid driving? Was it because somebody actually told us or is it because we got lost and we had to drive around?

Stefan (17:31):

Well, in on road driving, there's this metric that a bunch of states require of miles per disengagement. So obviously we should just track that.

Ilia (17:40):

Let's break that down-

Stefan (17:41):

As that gets better, we will be better.

Ilia (17:44):

Well, let's break that down. What does miles disengagement mean?

Stefan (17:46):

The number of miles your autonomous system drives before it has to turn off, easy-peasy.

Ilia (17:50):

And what do you mean by turn off?

Stefan (17:51):

It doesn't work anymore.

Ilia (17:55):

So I think in a lot of these systems, they're assuming there's a human driver or at least a tele-operated driver who's there watching.

Stefan (18:03):

Oh yeah. But once your miles for disengagement are a hundred, then you're safe for a two, four mile trip, whatever.

Ilia (18:09):

Yeah. Interesting case. I think we covered this tangentially, at least on the field or other podcasts. But the really interesting one about miles for disengagement is it makes no mention of what test scenario you're in.

Stefan (18:22):

Or what a mile is.

Ilia (18:23):

Or what a mile is, or what a disengagement means. So if I really wanted to game the system, I could put my robot on a perfectly flat circular track with no obstacles, with perfect GPS coverage.

Stefan (18:35):

Your metric will be great.

Ilia (18:37):

And you just run it and run it and run it, and your metric will be fantastic. Nobody ever interrupts.

Stefan (18:41):

Metrics are good, investors get on board and in turn make that the sweet profit baby.

Ilia (18:46):

Exactly. So it's not only a number, it's a number relative to something. It's a number relative to time span, to conditions, to the particular thing you're trying to test. Miles per disengagement sounds like a good idea but is in reality, pretty heavily gamed and not as representative as it should be.

Stefan (19:08):

Mean it also just doesn't really mean that much.

Ilia (19:10):

It doesn't.

Stefan (19:10):

Frankly, when I was going through YC with Star Ski, we were asked, our metric was how many miles are we driving a week? Well, at the time, it was an accomplishment to turn the robot on and drive in a circle. But after the first circle in the parking lot, the fifth was about the same as the fourth, as was about the same as the second. And we didn't learn things in those succeeding circles, one after the other. I think the challenge with measuring if the robot is robotting is essentially that what robotting is is very different on a robot by robot basis.

Ilia (19:47):

Yeah, especially if you have different clients that have different business objectives.

Stefan (19:50):

Yeah.

Ilia (19:51):

If you're trying to till a field versus if you're trying to move a mining vehicle, very different understanding of success.

Stefan (19:58):

Well, not even that. If you are trying to till a field and you are farmer Joe who's been farming for four generations, or if you're trying to till a field and you're a billion dollar ad company, and if you're trying to till a field and your tremble, there's all very different measures of what the number is that means anything to them. Because for a farmer Joe, it's like, yeah, it looks like you guys tilt the field. For the billion dollar ag company it might be, okay, so you tilt the field at a cost of this as opposed to that other cost, I guess this makes sense. And for a more sophisticated player like a tremble, like a bear flag, whoever, it's a, I don't know, your cross track error was greater than this. These sorts of models were firing off the wrong numbers too much of the time.

Ilia (20:45):

Exactly. Part of that is you really got to think about what is the audience for your metric as well. Is this a business focused metric like revenue, for example. Is this an engineering focused metric like cross track error, well, upon average over the last path. But the tempting thing to think of is if I can just think of the perfect metric, then if I see it go wrong, then I know that something's wrong. And realistically, that's very difficult if not impossible to do. What we've found is the best thing to do is have a bunch of metrics across different domains, and you have to track all of them to get a sense of how your system is performing. But normally that when things do go wrong, you have to correlate it to actual log data and sensor data and those kind of things.

Stefan (21:30):

What about metrics for safety? What metric means I'm safe?

Ilia (21:34):

Yeah. What sort of AI?

Stefan (21:40):

A question from a VC that we received in the last six months.

Ilia (21:43):

Yeah, that's great. I think the first person who could come up with a metric for safety would be very well off. It's probably not miles per disengagement.

Stefan (21:54):

Well, I mean, I think more broadly, an issue that we saw just the other day is a client has multiple cameras on their vehicle. And we saw a person walking around in the front right camera. They started walking towards the field of view of the front left camera and they disappeared.

Ilia (22:13):

Yeah, that's-

Stefan (22:14):

They were gone.

Ilia (22:15):

That's when we realized our cameras were not synchronized properly.

Stefan (22:19):

And the problem was this vehicle is a, I don't know, 20,000 pound vehicle.

Ilia (22:24):

Enormous.

Stefan (22:24):

It has mean looking claws. It can drive over any terrain. It is not something that you'd like to give you a little up tap when you're trying to parallel park.

Ilia (22:34):

Yeah.

Stefan (22:34):

So I guess in that case, Ilia, how do we use metrics or how can you use metrics to... I mean, can't we just know if that picture is out of date, if that's an old image.

Ilia (22:47):

When we were talking about this earlier, I was talking about photon to photon delay. That would be the ideal case. So what you do is-

Stefan (22:54):

And what'd that mean?

Ilia (22:55):

Yeah, exactly. So what you do is, is basically imagine a photon hitting the camera sensor. Then the camera processes it, sends it to computer, computer sends it to a network, network sends it to your laptop, laptop shows it on your screen. The photon comes out of the screen and hits your eyeball. What is the time from photon hitting the camera to photon hitting your eyeball? That's the ideal true latency measurement. It's also practically impossible to measure.

Stefan (23:20):

Yeah.

Ilia (23:20):

Because you got to have some atomically precise light emitter on one end and atomically precise receiver on the other end. If you can't do that which probably you can't, you got to step back to things like, does the sensor itself, the camera sensor does a timestamp the image it's getting? The thing that it's time stamping with, is it actually correctly synchronized? Is that camera getting some accurate time pulse from GPS or from an NTP-

Stefan (23:48):

And can we have essentially a watchdog on it that emits a metric?

Ilia (23:53):

That's even next step, right?

Stefan (23:53):

Yeah.

Ilia (23:55):

Yeah. First step is, is the image timestamped accurately or does the camera have the wrong time? Next step is there should be a note somewhere on the computer that's receiving this image, checking the timestamp and understanding is that time stamp valid versus the computer's time? If it is, how often is it coming in? If it's coming in too rarely or if it's dropped out, then you could say, "Oh, I've lost camera connection. I should probably come to a safety stop."

Stefan (24:22):

And these sorts of heartbeats and watchdogs, it makes sense whether you say it. Recently, a autonomous vehicle company crashed a vehicle specifically because they were going around a bend, they issued a steering command, the safety operator turned off the autonomous system, turned it back on 45 seconds later when the vehicle is now driving straight and the output command of, we're going around a bend or perhaps the perception of, we're driving around, a bend had not been properly timestamped and the vehicle attempted to steered right into a barrier.

Ilia (25:00):

Yeah.

Stefan (25:00):

And this is Robotics 102. I mean, technically not 101.

Ilia (25:05):

Yeah. What are the next step? The problem that a lot of software engineers make is they treat robots as slightly more mobile servers and they're not. Robots are existent-

Stefan (25:17):

Well, the servers with some gripper and stuff.

Ilia (25:20):

Servers with an arm, right?

Stefan (25:21):

Yeah.

Ilia (25:22):

But they're physically embodied, so they have to respect physics. They have to respect inertia. They have to respect those kind of things.

Stefan (25:27):

Physics is a conspiracy theory. It's not real, fake news. Isaac Newton was a hoax.

Ilia (25:34):

We continue our [inaudible 00:25:36] Conspiracy theme show instead of Robotics. But yeah. So the main thing, if I wanted to give one defaults, good, safe, best practice is timestamp everything.

Stefan (25:50):

Yeah.

Ilia (25:50):

Timestamp your messages coming in, timestamp your messages coming in-

Stefan (25:53):

Yep.

Ilia (25:53):

Have an extremely good understanding of photon hitting a camera sensor, all the way down to the wheels changing their motion because of that. So if you want to test this, your robot's going towards a corner. You throw a box in front of it. From the moment the box is visible on the camera to the moment the robot starts to break, to slow down, figure out what that delay is. And break it down and figure out what parts of your system are adding delays to that total value.

Stefan (26:21):

And how much of that delay is acceptable or how much of that delay is not.

Ilia (26:23):

Exactly.

Stefan (26:24):

Obviously because of physics and things like the speed of light, you can't see something before the photon hits you.

Ilia (26:28):

Yeah.

Stefan (26:29):

Obviously, you also can't break an 80,000 pound vehicle in one foot without significant heat being cost.

Ilia (26:37):

Or just tearing the ground apart. Yeah. I mean, speed of light probably will act the least on your total delay.

Stefan (26:46):

I think it's pretty passable.

Ilia (26:47):

Yeah. But your inertia will be a big deal. Big deal.

Stefan (26:50):

Yeah.

Ilia (26:51):

So you can correct this with sensors, they can see further away. They can see faster, so higher frame rate can help as well.

Stefan (27:00):

In terms of these metrics, I mean, something that people bring up a lot when it comes to safety is a deterministic system. If you have properly constructed heartbeats and watchdogs and just generally good metrics, can't you use non-deterministic things, deterministically?

Ilia (27:19):

Define deterministic?

Stefan (27:20):

So machine learning, for example.

Ilia (27:23):

Oh, I see.

Stefan (27:24):

If you say this is a lane detecting model.

Ilia (27:26):

Yeah.

Stefan (27:27):

There's a finite range that the lane should change in any 10 foot span. If I see a change bigger than that, reject it.

Ilia (27:35):

Yeah. In the ideal case, yes. But the reality is again, that machine learning models today have a very hard cutting edge explain ability of observability problem, where you can't actually explain why the model has made a particular decision.

Stefan (27:52):

Yeah.

Ilia (27:53):

And so because of that, you can't, in a safety sense, guarantee that the model will output the same value every time.

Stefan (27:59):

But you could say, I'm only going to listen to it if the range is within this particular area.

Ilia (28:08):

Well, range is in

Stefan (28:09):

The next expected value, is this close to last one.

Ilia (28:14):

Right.

Stefan (28:15):

Right. So I don't believe that humans magically disappear. So if I saw a human for five frames and I miss them on the sixth, then maybe I can say, "Huh, there's something weird going on here. We should come to a stop."

Ilia (28:28):

Yeah and that could be done but then you're wrapping your ML model in a bunch of classical deterministic mode, right?

Stefan (28:28):

Yeah.

Ilia (28:37):

So your ML model becomes one more input.

Stefan (28:39):

Yeah.

Ilia (28:39):

Which again, can work but there's a lot of complexity on actually ensuring that that's safety rated, safety passable.

Stefan (28:47):

Yeah. More just talking about measurable.

Ilia (28:47):

Yeah.

Stefan (28:50):

Yeah. Safety measurable more than safety passable. So let's say we have our system, we have metrics coming out of every corner of it to understand how well it is or is not working. How do we then start to look at things like overall performance over time?

Ilia (29:08):

What you really want to do is you want to understand, does any particular software change or commit that you've made or your team has made? Is it improving things? Are we moving in the right delay? So if you're measuring something like latency.

Stefan (29:21):

Yep.

Ilia (29:22):

Has this release improved latency or reduced it or made it worse? So you really want to have this good understanding of your system.

Stefan (29:30):

Or if we're measuring cross track error.

Ilia (29:32):

Yeah.

Stefan (29:32):

Is that getting worse or better, whether the same inputs are coming in and the same outputs are coming in.

Ilia (29:37):

Yeah and again, for anything sufficiently complex, very complex system, it's very unlikely you're going to get a commit that improves everything across the board. It's almost always some sort of trade-off. Like you reduce latency but now your accuracy is lower as you're using some heuristic or you're much more accurate but now it takes twice as long to process. So making these trade offs and looping back to your original design intent and your safety documents and making sure you can justify why this increase in latency, for example, is acceptable and that you're still within bounds. And sometimes the answers, it's not acceptable and you have to redo it.

Stefan (30:17):

Yeah. When we're thinking about what a general metric might be and how all those different things feed into it, I think it comes to a how reliably can we do the job that the customer cares about?

Ilia (30:31):

Yeah. And those are business post facing metrics. Again, you're going to have these full range of some metrics are facing your particular client, their use cases. Some are very pure engineering cross fair, for example, latency.

Stefan (30:45):

Let's come up with a particular metric for a particular industry, so we can stop just saying it depends.

Ilia (30:50):

Yeah, no. Okay.

Stefan (30:51):

As a global answer to all hard questions.

Ilia (30:53):

Yeah. Sorry, I'm getting a little bit too lawyer-y there.

Stefan (30:56):

Yeah. So what's a good use case for us to think. Well, maybe we could think about an office robot.

Ilia (31:02):

Sure. Back to our example.

Stefan (31:04):

And a less obnoxious one than the ball buster 7,000. Maybe we're looking more at just a telepresence robot. So we're a telepresence Robotics company. We want to compete against DOABLE Robotics and whoever else is in the space. I frankly, know one besides that. And we're talking to somebody and we want to say, "Hey, you should buy our robot over Doubles."

Ilia (31:27):

Yeah. I think from the business side, you'd be looking at engagements. You'd be looking at... What's up?

Stefan (31:32):

I'm laughing because it's hard to think of any real technical metric that that customer would ever actually care about, which is true and across the board here.

Ilia (31:40):

Yeah. Again, your business succeeds despite your robots.

Stefan (31:43):

Yeah.

Ilia (31:44):

Well, we can come up with some. There are some technical metrics. So business one would be things like how often is your system used? How happy are your people with your system? Those are classic web based stuff.

Stefan (31:53):

Yeah.

Ilia (31:54):

A technical one would be something like, how long does your robot run on a charge?

Stefan (31:57):

Yep.

Ilia (31:58):

How long is your battery life?

Stefan (32:00):

Yeah.

Ilia (32:01):

How many errors do your customers get when they use your webpage to interface with it?

Stefan (32:06):

To maybe talk a little bit more about the main types of areas that we're interested in.

Ilia (32:10):

Yeah.

Stefan (32:11):

A little bit higher stakes. Maybe let's talk about spraying chemicals in the field. So in that use case, you have a, let's say, autonomous tractor where autonomous self-propelled sprayer that is spraying chemical throughout the field. The ideal speed is that we're driving at two miles per hour, spraying some number of gallons per some unit of time, a gallon a minute maybe. I really don't know.

Ilia (32:38):

Sure. Your guess is as good as mine.

Stefan (32:40):

Yep. Yeah. How do we build the metric story of, this is a good robot, you should buy our sprayer over GUSS's.

Ilia (32:48):

Yeah. I would assume what you would care about would be things like, how many times has your tractor gone over a specific area? Because you really don't want to compact down the soil. You're going to damage it if you keep dry thin place over and over. So optimal path planners would be important.

Stefan (33:05):

Well, but that's only the optimal global path plant. Don't drive past the same vines over and over.

Ilia (33:05):

And global-

Stefan (33:05):

But-

Ilia (33:13):

And local even. So imagine you're spraying the same area month after month after month.

Stefan (33:18):

Yeah.

Ilia (33:18):

Don't want to control to exactly the same tire ruts you made last time.

Stefan (33:22):

Yes.

Ilia (33:22):

Or maybe you do, depending on what kind of crop.

Stefan (33:24):

Yeah.

Ilia (33:25):

You might want to shift intentionally your local path a little bit left, a little bit right, to not dig grooves into the ground.

Stefan (33:32):

But this all seems to me like a metric of farmer doesn't care about.

Ilia (33:37):

Yeah. The technical metrics. A farmer probably would care about how much spare are you using?

Stefan (33:42):

Yep. Are you reliably only spraying a gallon for a minute?

Ilia (33:42):

Exactly.

Stefan (33:46):

Are you reliably driving at two miles an hour when you're in the row?

Ilia (33:49):

I mean, I don't even think he cares about that. I think he cared-

Stefan (33:52):

Well, no, two miles an hour in the row. Sorry. Are you reliably only applying one cup of chemical-

Ilia (33:59):

Bingo.

Stefan (33:59):

Per vehicle?

Ilia (33:59):

Exactly.

Stefan (34:00):

And is the one cup you're applying mostly sticking to that plant.

Ilia (34:05):

Exactly.

Stefan (34:06):

At whatever mostly means?

Ilia (34:07):

Exactly. Yeah. Right. He cares about inputs or sheet. Cares about inputs of material and outputs in terms of crops.

Stefan (34:14):

So in other words, what we've figured out here is that if you're trying to build a robot for someone like a farmer, the better your robot is, you don't get much credit for. Your technical metrics don't really win you any deals or make you any more money.

Ilia (34:32):

Yeah. Especially there's a really spoofy ML based ones of, our model has this kind of error rate and is this accurate across fields? And that makes no difference.

Stefan (34:45):

I remember seeing some now publicly traded autonomy companies, series B decks where they said, here's our ML model. Here's how reliable it is. Here's its improvement over the last six months. Here's projected improvements over the next six. It's going to be amazing.

Ilia (35:00):

That's total nonsense. That is literally useless. Yeah. That will make no difference in the grand scheme of things. Unless your entire business is building effective ML models, that's not going to make a difference.

Stefan (35:13):

So let's say your business is just building a robot because this is again, a how to build robots podcast. So what are then the actual technical metrics for making sure that we can spray that field and that we actually are doing, let's say we're not competing against a competitor because our customers aren't going to care about our technical metrics versus competitors, but we just want to make sure the engineering team is effectively mushing and effectively making the robot robot better.

Ilia (35:39):

Building stuff. Yeah. So key tactical metrics there are things like, how long did it take to compute a path? Which generally is fast, but it could be quite a long time if it's very large.

Stefan (35:51):

A global path or a local path?

Ilia (35:52):

Both.

Stefan (35:53):

Yeah.

Ilia (35:53):

Both. How accurately are you keeping to the desired path? Did you cover the entire area efficiently or did you have to backtrack things like as you're driving, something you don't want to do especially with Rio vehicles, is continuously to jerk the steering back constantly.

Stefan (36:11):

Yep.

Ilia (36:11):

So your acceleration and jerk parameters over your path-

Stefan (36:17):

How not nice your robot.

Ilia (36:18):

Exactly. We have not nice both visually and analytically. A lot of customers, it's funny I've noticed, because they're seeing a robot and they don't understand the detail of what's gone into it, they're going to judge the robot by how confident it looks.

Stefan (36:33):

Yeah.

Ilia (36:34):

And if your robot's constantly nudging, steering back and forth-

Stefan (36:38):

Looks like you got the robot.

Ilia (36:39):

It looks terrible. Even if that's technically more efficient.

Stefan (36:42):

It almost makes you think maybe there's some folks who built robots that only update their path every 10 meters because they're not good at controls but it's a simpler way to make it look like they are.

Ilia (36:53):

Look like it's smooth. Exactly-

Stefan (36:53):

Yeah.

Ilia (36:55):

Yeah. What else? What are the technical metrics? Just basics like how as your software gets more complex, crashes.

Stefan (37:03):

Yeah.

Ilia (37:04):

Just like in recoveries, how often are you-

Stefan (37:06):

Do you mean crashes as in collisions or do you mean crashes as in the system no longer is up and running?

Ilia (37:12):

Both. Realistically, in a complex system, you don't have just one software node, you have a bunch of them.

Stefan (37:12):

Yeah.

Ilia (37:18):

And so is any particular one of them faulting for some reason? Yeah.

Stefan (37:22):

So metrics are pretty obviously important, not just because I want them on the dashboard, but primarily because that's how we know if our system's actually improving or not. I'd expect then that the metrics ecosystem in Robotics is incredibly well built out. There's great tooling. Everyone uses the same standards for the tooling, even though there's some, this metrics' good for this company, that metrics' good for that company, right?

Ilia (37:46):

Yeah. I would love that to be. That would've been wonderful. That would've saved us a lot of effort. Unfortunately, we're pretty heavy ROS and ROS2 users, and in the ROS community at least, there's not a lot of focus on this. We've started to publish some stuff.

Stefan (38:03):

Why not?

Ilia (38:04):

That's a great question.

Stefan (38:05):

Yeah. Probably because there's not that many real robots.

Ilia (38:07):

It might be because they're not that many real robots. It might be because people don't see the benefit of this. The most common question, even in our engineering team when we introduced it was the question of, "Well, isn't this just log messages or ROS bag files? Why not just Ross bag this? Why would you need metrics?" That approach 80 doesn't scale to tens of thousands or millions of robots because ROS bags from a autonomous vehicle are terabytes in size. Each can't sling that much data, but secondary-

Stefan (38:36):

That you just don't want hang out with the ROS bag. You don't want to be called a ROS bag. You don't want to be a ROS bag.

Ilia (38:41):

We have team shirts. I'm sure we'll sell them someday.

Stefan (38:44):

Yeah.

Ilia (38:44):

Just called Total ROS bag.

Stefan (38:46):

On our merch store.

Ilia (38:47):

Yeah, exactly. On our future merch store. Thank you to our wonderful producer, Nicole, for designing and putting those together. But even more importantly, imagine yet all the compute and storage and networking in the world. As an engineer, I don't want to sit through three hours of ROS bags trying to find the 30 seconds where my thing went wrong.

Stefan (39:06):

Yep.

Ilia (39:07):

What I want is a system on board that automatically flags my ML model confidence drop below a certain value, or I detected a collision or whatever.

Stefan (39:17):

Yep. Absolutely.

Ilia (39:19):

And correlate those metrics with your more rich ROS bags of log files and those kind of things.

Stefan (39:25):

Or with other metrics that might be leading indicators that there's going to be a problem.

Ilia (39:28):

Exactly.

Stefan (39:28):

Maybe every time we need to come to a sudden stop, a hundred milliseconds beforehand, a human randomly pops up on the screen.

Ilia (39:36):

Yeah.

Stefan (39:36):

We weren't expecting, and that happened because we stopped getting messages from a camera for 200 milliseconds.

Ilia (39:36):

Exactly.

Stefan (39:39):

And one could say that if when you stopped getting messages from my camera for 200 milliseconds, if a watchdog saw that and ordered an emergency stop, you might've been slightly safer.

Ilia (39:53):

Yeah.

Stefan (39:53):

One could say.

Ilia (39:54):

One could say tonight. But yeah. One final note on this, A trick that we put on our blog post and on metrics-

Stefan (40:04):

Well, we should share about the fact that we have a blog post.

Ilia (40:07):

Yeah.

Stefan (40:07):

I need some open stuff, open source.

Ilia (40:09):

Yeah. So we do have blog post on our website on metrics and ROS bags by very talented intern David who’s happily returning.

Stefan (40:17):

Yeah.

Ilia (40:17):

Wonderful.

Stefan (40:18):

Mostly because we kidnapped him.

Ilia (40:20):

But basically.

Stefan (40:21):

Yeah.

Ilia (40:22):

Yeah. That was the effort.

Stefan (40:22):

No one here works willingly. Please let me go.

Ilia (40:27):

So yeah, so he had a great blog post about that. In there, we talk about how we correlate metrics sin in more rich bake fells. But what final trick to mention here, is a lot of times when you have a problem with a robot, when it hits something, the actual thing that went wrong happened a minute ago or 30 seconds ago. And so what you really want, is you want a rolling buffer of data onboard that is storing this terabytes per second generated, but you don't want to store all of it to desk. You'll run at a disc in a day or less. So what you want is you want a rolling buffer that only actually stores to disk when a metric detects or when a metric fires like a collision or ML certainty or whatever. That allows you to get the best of both worlds. Both really rich data, but also not hours and hours of it.

Stefan (41:15):

Yeah. And that's how you get to a world where you can make robots robot more faster.

Ilia (41:21):

I couldn't have put it better myself. And on that super eloquent note, I think we'll close this one out. Next week, we're going to talk about making a business case for your robot and how to actually build a business other than despite your robots as we connect.

Stefan (41:37):

Primarily kamikaze flying your drone into your employee's crotch. That's number one business.

Ilia (41:44):

Yeah.

Stefan (41:44):

We'll see you all next week.

Ilia (41:47):

Signing off.

Stefan (41:47):

Space Cowboy.

‍