Ep 3 - Teleoperation and Art
Notes: this transcript has not been edited for clarity or mistakes.
Stefan (00:03):
Hello and welcome to Automate It, our weekly podcast where we talk about building robots. I am Stefan Seltz-Axmacher, the name that is so short, so memorable that you absolutely know how to pronounce it every single time. With me is my co-founder and co-host Ilia Baranov.
Stefan (00:03):
Hello and welcome to Automate It, our weekly podcast where we talk about building robots. I am Stefan Seltz-Axmacher, the name that is so short, so memorable that you absolutely know how to pronounce it every single time. With me is my co-founder and co-host Ilia Baranov. Ilia, could you speak or do I just speak for you as CEO?
Ilia (00:23):
Well, I'm just the puppet master behind the strings.
Stefan (00:25):
The strings seem very frayed.
(00:38):
For those of you who are just joining us for the first time at Automate It, we talk about building robots, how we've built robots before. And we start off with a game that I really love where we draw two cards out of a deck, one of which has a technology on it that we either have to fit into a solution or that we might. The other is a setting. And the way to think about this is to imagine two co-founders teaming up, where one of them has some business insight into a particular setting or a market, and the other is a CTO who has some technology they're desperate to use. Or sometimes absolutely would despise and never use. There's no point of it. It's super over-hyped. Ilia, why don't you draw the setting card for us?
Ilia (01:21):
All right. Let's see here, we got senior care.
Stefan (01:26):
Senior care. Man, so as a disclaimer, my wife works in senior care. I have talked to a number of folks about robots in senior care and this will be fun. And we have on the other side, teleoperation.
Ilia (01:40):
The classic mix.
Stefan (01:41):
Yeah.
Ilia (01:43):
Amazing.
Stefan (01:44):
I think there's a couple ways that we could go here. I mean, the most serious could be senior care robots that are teleoperated from some lower labor cost place, that seems like a real business. Or, better than that, I think we could look at every five minutes, we talk about the end of social security in America. And these seniors, they go putzing around working 60, 80 hours for their whole life. They decide at 75 they want to retire and enjoy a fleeting couple of minutes with the grandkids. But they're no longer working and if you're not working, I don't know, what's the point? So why don't we use seniors as teleoperators?
Ilia (02:24):
I was going to say, we should flip the script.
Stefan (02:25):
Yeah.
Ilia (02:26):
I was totally thinking, I wasn't thinking quite so capitalistic hellscape, just thinking more along the lines of maybe vacations for people who have reduced mobility with kind of a VR standpoint.
Stefan (02:39):
Yeah, yeah.
Ilia (02:39):
Let's go with the capitalist hellscape.
Stefan (02:41):
Yeah, let's go the capitalist hellscape. I mean, I think the thing is, so for those of you who don't know, life in a senior care facility is really expensive. So when my wife first moved out here to San Francisco, the senior care facility that she worked in was charging, I want to say eight to $10,000 per bed per month for an assisted living facility. And that's not a place with nurses and doctors. That's a place where there's a nurse's assistant who hands you the medicine and they play old polka on the stereogram.
Ilia (03:11):
That's why cruise ships are so popular. By comparison they seem quite cheap.
Stefan (03:16):
So if you think about it, let's say between your family wealth, your life savings, you have a measly $600,000. That's not going to last you very long if you intend to not croak in the next three to four months. So these seniors, they need to teleoperate. How else are they going to pay their denture bills?
Ilia (03:36):
It's going to be an interesting case because you need a task which requires a human touch, but doesn't require really quick response rates.
Stefan (03:47):
So to be fair, I mean teleop's never really that good in a high response rate. It's like no matter what you're going to see a hundred milliseconds of latency. So yeah, maybe you can drive that up to 20, 25 miles an hour, but it's never really steering the motorcycle around the racetrack at 80 miles an hour. So I feel like actually the seniors aren't going to be that taken away from, that disabled, by having slower reaction time than a 15 year old teleoperator in Paraguay.
Ilia (04:20):
Yeah, that's true. I think the other problem though is the inherent nausea with any kind of VR system or any of that stuff.
Stefan (04:29):
VR systems, honestly, when we did teleop at Starsky, we found that drivers ended up feeling nausea, having trouble feeling a bit disoriented after two, three hours of really high stakes, high speed teleop. So my guess is it's not so much a high speed, you have to be on all the time type of teleop scenario. It might actually be more of maybe autonomy gets you, maybe it's say a robot in a hospital or a robot in a hotel, places where people love the idea of robots. And then you have a senior who's telling people how to get her out of a hotel.
Ilia (05:07):
It could be that. It could be, even in that case, what you're really getting at I think is human supervision of automated tasks. So what you really want is you want kind of human judgment to come in once in a while and most of the time the system is operating itself. But once in a while it gets stuck and you need a human to come in and say, "You know what? Yes, you should do this now because the door is jammed. Go call for help."
Stefan (05:32):
I do feel like we're getting closer and closer into real business. Because now that I think about it, I remember hearing a teleoperated robot where the pitch was seniors. That's deliberately not part of this segment. It's not supposed to be as serious.
Ilia (05:46):
No. Yeah. Let's get back to capitalist hellscape.
Stefan (05:48):
Yeah, capitalist hellscape.
Ilia (05:50):
So being Canadian, American politics are pretty interesting to me.
Stefan (05:56):
Because they're perfect and ideal in every way.
Ilia (05:59):
Absolutely. So I mean with all the voter intimidation that's been happening in some states, we could have predominantly right leaning seniors automate their voter campaign.
Stefan (06:15):
I mean, yeah, we can just take the bomb disposal robots and anywhere where you're say, just to make up a thing that would never happen, anywhere where say it's illegal to hand out water to people waiting in line to vote, you can just have the bomb disposal robot mow people down.
Ilia (06:30):
Actually that's a good point. Is it illegal for people to hand out waters?
Stefan (06:37):
I don't know. I'd have to check the Georgia, I mean Georgia, the country, statutes on who can or cannot hand out water. And I think actually we're opening into an interesting area. Let's say you make it the chat roulette of teleoperation where the teleoperator is paired with a random device for say until they want to pick away. You'd have kind of complete plausible deniability of whether it was your fault or somebody else's that whatever you did. So you could interact with the physical world and then just throw your hands up in the air and blame, I don't know the blockchain that assigns you to that particular robot and note it down for no.
Ilia (07:17):
It's got to be a blockchain. It's always...
Stefan (07:18):
You need the block, this is a great use case for blockchain.
Ilia (07:21):
I heard this really great coin FTX, we should put one hand in for this endeavor. Yeah, no, less accountability for teleop.
Stefan (07:31):
Actually. I mean coming back towards something that might be real but still not, is living vicariously through robots. So not so much like a VR type of headset thing of go see Paris. What I mean is again, bomb disposal robot drives around the elizy whatever, really freaks out a lot of people, including ironically probably some other seniors while all some blue hair in Florida is having the time of her life doing the trip she never got to do.
Ilia (08:02):
Yeah, yeah. And then there there's a full American, the army helps you get where you want to go in life kind of angle that we should think about.
Stefan (08:10):
I think the VA actually. So not a lot of people want to sell weaponized robots to the military. Maybe the way that you breach that subject initially is you can sell non weaponized robots to the VA and give old veterans the ability to travel around the world on treaded vehicles. They're pretty light, they only weigh like one to four tons. For drive around the sidewalk of Europe and Georgia and anywhere else that you're interested in.
Ilia (08:36):
Bring freedom wherever they go. Beer. Yeah. Oh, that got dark. What other teleoperation case could you have?
Stefan (08:47):
So to talk about the serious one, people who work in senior living facilities are actually incredibly hard to staff. This is a hard job that is relatively thankless. Many of the companies that run the senior living facilities are real estate investment trusts with a couple of extra wrinkles around it. So they end up being very bottom line focus. They hire a bunch of blue collar workers. They're paying somewhere between minimum wage and top end retail wages. So let's say in between a McDonald's and a Best Buy. But it's a job that you need special licensing for. When my wife was managing a bunch of people who had this job, many of them would actually have two full-time jobs. They'd be working like 60, 80 hours a week and seeing their senior care job as the way they were giving back, because many of them were from cultures that put a lot of value in taking care of seniors and doing things like that.
(09:45):
So I think if you were willing to make it a cross-border teleop operation system, you really could have people from way Bolivia, Ecuador, Peru, who would just be stoked to give back to an older generation, would be really happy to make the 6 or $7 an hour that system could pay them probably. Because let's say the system's earning 16, $17 an hour, 4 to $6 is profit, the next 6 is for the robot, the next 6 is for the teleoperator. And that's hard. So that adds up to more than $16. But you would have a system like that and end up having not senior care that could solve all problems, but probably you could do some stuff that people can't currently do and the in-person caregivers could spend more of their time focusing on face-to-face connections and adding face-to-face value.
Ilia (10:44):
I mean think the thing you got to think about is how are you defining care, right? Because if you have to do any kind of manipulation task, you immediately blow the cost of the robot. If it's a screen on wheels, it's a few thousand bucks in production.
Stefan (10:58):
A double robot.
Ilia (11:00):
If it's anything that has to pick stuff up, it's immediately a hundred cut.
Stefan (11:04):
Well, I don't think I'd want a robot arm capable of moving engine blocks picking up gram, but I don't know if that is quite interesting because grandma is a little frail and with whatever, 800 pounds per square inch torque, whatever, grandma's not doing so well if there's a bit of controls issues and there's always controls issues.
Ilia (11:27):
No, but I mean even just picking up basic stuff, bring me a glass of water sort of thing. But you could use it as a force multiplier where you still have a few humans around, but you have kind of gopher bots.
Stefan (11:39):
Yeah, I mean you can use them as gofer bots. You could use them as cameras on wheels to check in on people and see if they're doing well. Then as a result you do check-ins on an hour basis, whereas right now it might be three or four hours because a lot of bad things can happen in the extra two to three hours.
Ilia (11:52):
So I feel pretty reasonably people will be uncomfortable with robots checking in on them. I don't know, I'd really love to hear listeners, if anybody out there wants to reach out to us, how has even just telepresence taken off with the senior care kind of case? Because is it very used to, assuming somebody sets it up for you so you don't have to worry about it, it just works.
Stefan (12:17):
I mean, I see ads from Facebook who would never lie to me, that the Facebook telepresence thing always has a picture of a senior talking to their grandkids. And if it's in an ad, it must be true.
Ilia (12:29):
I mean, Facebook only ever lies about election related stuff.
Stefan (12:34):
Ilia getting more than a little political today. I mean to that end, I know many people who in COVID, like seniors very much quickly grabbed onto things like Zoom, things like whatever. In fact, my mom who is now technically a senior, got very into Zoom very quickly and had all the nice things to say about it of, "Now I can talk to this person. Now I can talk to that person."
Ilia (12:58):
So really what you're saying is you want the telepresent system where seniors can visit each other. That sounds like a thing that exists.
Stefan (13:08):
Well, no, more of my point is that with really good robots, and I'm not saying really good like C3PO, I'm saying really good like a double robot with a iPad on a set of wheels that kind of makes it so that you don't have to learn a new UI, actually have to learn how to use a thing. There's just suddenly a face looking at you and there's a camera attached to that face and you can talk to a person. I think that is not too terrifying, not too dystopian and means that that senior is going to see a lot more faces. They're going to have a lot more human interaction. And frankly that if the person on that particular iPad wants to spend in seven minutes talking to the person in that particular room who either has a problem or just wants to say something or whatever, they can do that without being told they need to hit up the next 25 rooms by the end of their shift, which is supposed to be 15 minutes from now.
(14:05):
Because I imagine it doesn't feel nice to be in your golden years, try to tell an interesting story to this nice guy who works in the facility that you live in and him saying, "Yep, yep, yep. That's so great Mr. Stefan, I need to leave because my kids also want to see me. So unless you are having an episode, I'm out of your room now." Whereas if that guy's shift could end after our conversation and someone else could be on another double robot to check it on the other 15 seniors, that might be kind of a best of all worlds.
Ilia (14:40):
And I think the best way to prototype it is with these bomb disposal robots.
Stefan (14:47):
Just batch them. The first folks are going to be chew robot friendly. So you have to make sure that the robot's tough enough to survive.
Ilia (14:55):
That's right. Survival of the robot friendliest.
Stefan (14:59):
I'm just going to say the food at my grandmother's senior living facility was not very good. Think bombs is pretty close to it.
Ilia (15:06):
Oh God.
Stefan (15:08):
With that I think we'll transition to talk to eat some real robots that we worked on. While we've probably alienated all of our listeners over the age of 74, I'm very sorry, Gladys. I'm very sorry, Ethel and Mr. Johnson, I know we're not on the first name basis yet, but I hope you keep our listening to the podcast.
(15:27):
To move on from there Ilia, what we're going to talk today about a robot that you worked on that I always thought was a really interesting idea.
Ilia (15:35):
Yeah. We affectionately named it Fan Gogh after the Fanook carb, was it?
Stefan (15:41):
To zoom out before we get to the naming part, which is the most important part of any robot, I guess to play as similar as we play our currents. What was the point of this overall project? What was the set of technologies you guys were looking at using and then let's kind of work our way to what you guys built.
Ilia (15:58):
Yeah, yeah. So this was a trade show machine, which at Clearpath for trade shows, we've always wanted to have a nice little demo in this particular year. What was time off was-
Stefan (16:08):
To dive into that a little bit, a hard thing about robots as we both know is these things don't work all the time and the most valuable ones look pretty boring after the first five seconds. So building something that is actually demonstrable that fits in a trade show. And also when it comes to what you could do in a trade show, I know how much time you've worked, a lot of shows when you're running around the world in Clearpath, the average group of people who walk up to your booth, how long are they there for?
Ilia (16:36):
You got maybe anywhere between 10 seconds if they're walking by and just want to say hello and up to five minutes if they really want to figure out what's showing.
Stefan (16:44):
So probably a three mile per hour drive around coastal California is off the table.
Ilia (16:50):
Yeah. Probably won't get you the response you're going for.
Stefan (16:55):
Okay. So the problems that then becomes what can you package up that is visually interesting, fits with it... I don't know how big is it 10 feet by 10 feet?
Ilia (17:05):
That's about the scale. You can also get a double 10 by 20, but generally 10 by 10.
Stefan (17:09):
Okay. So yeah, a hundred square feet. You need to catch someone's attention in 10 seconds. Ideally someone who doesn't have a PhD in Robotics, because whatever trade show you go to, there's not many of those.
Ilia (17:24):
I think the Robotics trade shows that we went to by ratio of Robotics species, I think that year the trend had been with the success of Universal Robots, which makes a bunch of human collaborative or cobot arms, a lot of other players were looking to get into human collaborative space. And so at the time-
Stefan (17:44):
Wait what's that mean, human collaborative?
Ilia (17:46):
Robot. Yeah. So that basically means that your robot won't kill you, first of all.
Stefan (17:50):
Well it's totally good. I've heard Asimov not pro robot killing.
Ilia (17:56):
Yeah, definitely the easiest thing to code up is Asimov's laws.
Stefan (18:01):
I just don't know why you can't work on a positronic break. I've been asking this. I've been doing a request for developers. No one seems to have positronic brain experience.
Ilia (18:09):
It's in our backlog.
Stefan (18:11):
Doesn't make sense.
Ilia (18:14):
But more than that, what you want is a system that will interview and that makes a little bit trickier, especially for arms. One is you just get a lot of kinetic energy of theorem spinning around and it's about to hit someone.
Stefan (18:24):
How hard can an arm hit you before it's an ouchie?
Ilia (18:28):
I don't know the exact number, but-
Stefan (18:30):
I was hoping for a three.
Ilia (18:33):
The main problem between a human bumping between you and an arm is that the arm is made out of steel, it tends to move quite a lot faster.
Stefan (18:42):
I don't know many people made a steel.
Ilia (18:44):
Actually side note, but a funny problem with using robot arms for dealing with cell culture is robot arms accelerate so quickly that they kill everything they're trying to move in the little Petri dish. That's really what makes them dangerous as well, is that their acceleration is so high that they get a lot of kinetic energy really quickly in a tiny amount of space. Their hardness combined with their speed, pretty dangerous. But that's only the first step. The second thing is you have pinch points and so you really don't want a finger caught between two joints where it'll get crushed. Because even at low speeds you can easily do that.
Stefan (19:20):
So the trend of the time was human robot collaborative things. You want it to be visually stimulating relatively quickly. It's supposed to be an arm. I guess what are the different ideas? I think of, you always see things like robots playing ping pong. Recently a company was founded out of MongoDB, founder Elliot Horowitz's desire to play chess with a robot arm. What were the early ideas you guys were interested in?
Ilia (19:47):
So in particular CorPath about was working with Fanook who had just released kind of one of their robot arms. They're a big manufacturer in robot arm space primarily. They make arms for industrial use cases. Really heavy, really big-
Stefan (20:00):
Robots [inaudible 00:20:02] and grandma.
Ilia (20:02):
Yeah, exactly. To our earlier discussions, they make really fast stuff. So this is their first, maybe not their first, but one of their early approaches to human collaborative robots. Kind of fun note, the way that they did it is they didn't want to retrofit the arm itself because that'd be expensive and would take a lot of design. So instead of that, they mounted a really accurate force sensor under the arm. And at any given moment as the arm moves, it would predict how much force it should see at the base. And if it saw more or less than that, it knew that it was doing something wrong and it would stop.
Stefan (20:36):
So it's basically a doer, checker architecture for speed of movement?
Ilia (20:40):
Yeah, for inertia, more than speed of movement is for inertia. So that's an interesting case where instead of trying to put force sensors at every joint, we had a big one at the base. It was kind of a clever solution because they don't have to make a new skew. They only had to basically change the paint color from red to green to indicate a cover.
Stefan (21:00):
Green's a great green.
Ilia (21:04):
So we knew we wanted to use that. That was kind of our partner for the year. And the other thing we wanted to use was the Ridgeback base, which at the time was relatively new. Kind of an omni-directional base, kind of stick the two together, what's something interesting we could do. My general manager at the time really wanted to drive around the robot, have it hit symbols to let people know that it was time for the next talk. That seem-
Stefan (21:24):
Driving around the trade show floor.
Ilia (21:28):
But that didn't-
Stefan (21:28):
Driving around the trade show floor with something that looks like a hammer. That's one way to get through [inaudible 00:21:34].
Ilia (21:34):
Very, very safe.
Stefan (21:36):
And so one of the things was hitting symbols. What else was there?
Ilia (21:38):
The other one, which we ended up going with, was actually having it draw on something.
Stefan (21:43):
I mean that thing is pretty easy. I remember watching Bicentennial Man, I recall Robin Williams as a robot drawing things. He was pretty fast at it. How hard can that be? Were you just drawing figure eights or my favorite, the S sign that we all grew up with?
Ilia (21:58):
Yeah. Hey, I started off the project thinking that I could make it draw realistically and that was quickly, classic engineer, [inaudible 00:22:08] is art, whatever. No, very hard. So I really wanted to drive around the conference room, take photos of people and give out a little business card with their drawn kind of photo and our little logo. Start off the process saying, "Okay, let's draw people." And then I realized turning a photo into a series of motion, it's pretty hard.
Stefan (22:31):
Well, yeah, I guess actually I'm interested in that, they have a trial. I guess what tool do you use to even turn that to what this drawing should look like? Let alone the motion planet. Is there a tool that's matrix of 5I face, but turn into shitty line drawing?
Ilia (22:53):
Yeah, I wish there was. I tried a few different approaches. Again, I thought it would be easy, but first approach I did is I took the photo and I did edge detection, standard machine vision tool, kind of look at differences in color-
Stefan (23:09):
What's the end of the person?
Ilia (23:10):
Or where's their features on the face? So any edge, right eyes, edge of eyes, edge of nose, edge of ears. Okay, so great. I have these edges. Now how do I draw them? So now I have a 2D image of pixels of black or white, either edges or no edges, but I still don't know how to turn that-
Stefan (23:29):
These drawings were just a very light outline.
Ilia (23:32):
Well, that's what I thought. That was my first attempt. But then I still realized I haven't actually made my problem that much easier because now I have edges, I have less data, but I don't have data to still turn this into motion. So my second attempt, I luckily stumbled onto a really fun library that would represent an arbitrary photo as a collection of different colored triangles. So it's basically turning it into a mesh and so it can build an image and transfer it into a SVG file, which standard internet file for vector graphics. And the nice thing about vector graphics is it actually defines, it tells you, here's a line and here's how big it is, which is something that's starting to approach motion planning. And so this file would take a photo, turn into triangles. Great. But the triangles, the way that it built the photo was actually filled in colored triangles, which if I told the robot arm to fill in a triangle with a pen would take three hours per photo. So I took that, I made it draw just the outlines and it ended up with a roughly impressionistic photo of the person.
Stefan (24:42):
And say if someone wanted to Google and find these photos and what they look like, what would they search for?
Ilia (24:47):
Yeah. I think it's still on the Clearpath Robotics product page or blog page of Fan Gogh. Sure you could find it. There's a little video of it doing a high speed sketch of me. But yeah, once I landed on this impressionistic style of these triangles, I had my starting and ending point for each triangle. I could translate that 2D plane into motion plans for the robot. And then the Fanook software was good because I could just say, start and end at these locations in 3D space. And it could figure out how to move the arm to do it.
Stefan (25:16):
Well I guess also, what drawing tool was it using?
Ilia (25:19):
Just a pen. There was a pen on the end of the arm and it would tell Fanook apply a certain amount of pressure on this plane and move through-
Stefan (25:27):
I guess how tightly controlled was by this much pressure? Because I feel like the safety things that would stop you from breaking someone's arm are still tolerant of snapping the head off a pen.
Ilia (25:41):
So we cheated a little bit, that's always the rule for good demos is you cheat-
Stefan (25:45):
Or just for robots. Robots is the cheating industry. Don't tell your spouses it's the only way to go.
Ilia (25:54):
So basically what we did is the pen was mounted on a very small spring. And so we always had a little bit of leeway. And so the spring would always apply kind of a constant pressure and the arm could be off by three or four millimeters and the pen would still touch the page.
Stefan (26:09):
I wonder how many people have this picture framed in their house.
Ilia (26:12):
Hopefully everyone at that conference. So I worked, I got it down to about three minutes per person, which was within that kind of attention span.
Stefan (26:23):
And let's talk through that pipeline. So what year roughly was this -ish?
Ilia (26:28):
2015.
Stefan (26:30):
Okay, so 2015. Maybe have a point gray camera or do you have what?
Ilia (26:34):
Cheapo USB webcam.
Stefan (26:36):
Okay. So cheapo USB cam is mounted somewhere just in the booth?
Ilia (26:42):
Nope, right on the robot. The robot would rove and take photos of people.
Stefan (26:45):
It was actually roving taking, that's cool of itself. Okay. What was it following a person with an offset?
Ilia (26:52):
No, it would randomly walk until somebody would see the little placard on it that said, "Press this button to get a sketch." And then people would press the button and that would trigger the operation of take a photo, look for the face using standard machine learning face tools.
Stefan (27:07):
So all this was on a mobile robot, battery powered?
Ilia (27:09):
Yeah.
Stefan (27:10):
Okay.
Ilia (27:10):
So the robot would rob around the conference and it would take photos.
Stefan (27:16):
What's the base of the vehicle again? It's a Clearpath-
Ilia (27:19):
Ridgeback, omnidirectional robot.
Stefan (27:22):
Okay. Well then what were the dimensions of a Ridgeback? How big is this robot?
Ilia (27:28):
Four feet by four feet, probably two feet tall.
Stefan (27:30):
So you wouldn't want to share an elevator with it?
Ilia (27:32):
You could squeeze in.
Stefan (27:34):
Yeah, but it wouldn't be comfortable. It wouldn't be a friendly elevator companion. So it's a four foot by four foot omnidirectional robot. It's got an arm. Does it have anything else? A beret, does it maybe have an accent that makes me feel like it's doing me a favor by drawing me instead of me as the art, the inspiration? Any other pieces of a flair?
Ilia (27:53):
We did 3D print one ear, from the inspiration Van Gogh. So there was a 3D printed ear stuck on the side of it I remember.
Stefan (28:06):
Did that get decided, did people notice that or did that get?
Ilia (28:07):
I don't think people noticed it. I think maybe one or two people got the joke, but it was kind of a small detail.
Stefan (28:13):
It would've been better if after drawing a particularly stressful person it had just ripped one of its ears off and hocked it at the person. As we discussed earlier, a robots thorough things at people are probably-
Ilia (28:24):
Probably a bad idea.
Stefan (28:26):
Yeah. Not a good look. Okay. So this free roving four foot by four foot robot has an arm. It has a cheapo USB camera on it. It's roving around. What speed are we talking? Half a mile an hour, mile an hour?
Ilia (28:38):
Very slow. It has two lidars on the base and it was a very base navigation. Get from point A to point B and very, very slow just because there wasn't really a case for having it drive around quickly.
Stefan (28:51):
And was there a person walking alongside it, or?
Ilia (28:51):
Yes.
Stefan (28:54):
Oh, okay.
Ilia (28:55):
Generally there was a person there to nanny it. Honestly, I don't remember if anybody had to intervene, but I'd be surprised 'cause it was just doing very basic back and forth motion.
Stefan (29:03):
So it had a sign that said, "I'm going to draw a picture of you," and the average person who was probably responding, "Only draw me wearing this diamond necklace."
Ilia (29:13):
Crossover between Bango and Titanic.
Stefan (29:17):
So a user walks up, pushes a button, how'd you target the camera at that point?
Ilia (29:23):
So there's a little display that would show what the robot would see so you had to get in the frame. Yeah, use the humans wherever you can.
Stefan (29:31):
Another cheat code. Why not, just to talk to that decision there, why not just have the camera point itself at the person?
Ilia (29:39):
You could, the fun thing about this one was the algorithm I was running for detecting the face to draw it was a very, very basic face detection algorithm. And in a crowd it would very likely pick out somebody who's not the face that you actually want. So I added on-
Stefan (29:55):
And it's bit more ominous if the robot draws a picture of somebody-
Ilia (29:59):
Somebody else.
Stefan (30:00):
Yeah. Go after this one, yeah.
Ilia (30:03):
Exactly. So very simple heuristic, it would just pick the biggest face it saw on the scene.
Stefan (30:07):
As a person with a very large head, I feel like I would be-
Ilia (30:11):
Personally targeted?
Stefan (30:12):
Yeah. Personally targeted, drawn frequently. I don't know if I feel like I'd prefer that. So you click the button, it targets the biggest head and then it immediately runs the face to triangles?
Ilia (30:27):
Yeah.
Stefan (30:27):
Okay.
Ilia (30:28):
Yeah. So it runs this open source, I forgot the name of it honestly, but it was an open source little SVG generator that then would desaturate, pick the top 30% biggest triangles and then draw out those triangles.
Stefan (30:41):
I mean, did it look much more real with all the triangles or why'd you pick the 30% biggest triangles?
Ilia (30:47):
Because I wanted it to finish in a few minutes. I didn't want it spending hours just-
Stefan (30:52):
Did you see an increase in how good the picture looks at more triangles?
Ilia (30:56):
Oh yeah, for sure. It got more and more realistic. I mean it never got picture perfect. It's kind of an impressionist sort of painting.
Stefan (31:04):
People don't say that we are made of but triangles.
Ilia (31:06):
Yes.
Stefan (31:08):
Okay. So pick the first 30% to stay within that five minute time window. So now you have your drawing. What's the next part of the pipeline?
Ilia (31:15):
So the next part is now have SVG, which actually is a file format that tells me start end points for a bunch of lines. Then I translate that to lines to actually coordinate some 3D space for the arm to move up and down. I forget actually if I mounted the, yeah, I think I mounted the easel it was drawing at, 'cause it was a fixed easel, a little bit on an angle.
Stefan (31:38):
Okay, why?
Ilia (31:40):
Just so it looks nice. And then I just had a little bit of math to translate just up and down, left and right motion to a little bit of depth motion as well.
Stefan (31:48):
What was the full range of motion of the arm? I imagine you just limited it so it wasn't going drawing triangles on people's real faces.
Ilia (31:56):
Yeah, no attacking real humans. We had it had to stay within a roughly one foot by one foot box.
Stefan (32:03):
And I imagine that one foot by one foot box is in the middle of the Clearpath vehicle, meaning that nobody could physically walk within it. Exactly. Also bad to draw on people's legs, they don't care for that.
Ilia (32:15):
Yeah. But I mean the bigger risk is when you're doing arm motion planning that there's this idea of inverse kinetics. So that's basically where, I know where the end of my pen needs to be from my drawing, but I don't know exactly what each joint of my arm needs to do.
Stefan (32:32):
I don't know about you, but I always think about my joints when you're drawing and moving.
Ilia (32:36):
Yeah, exactly. As a human, if you reach for something, you don't think, "I'm going to move my elbow 10 degrees." What you're thinking is, "I'm going to reach this point," and your joints kind of figure it out.
Stefan (32:45):
I mean, that's where the compute is.
Ilia (32:47):
Well that's what the brain's doing. So the equivalent to that for Robotics is inverse kinematics where your endpoint and you kind of solve the problem of where your joints seem to be. The problem is with inverse kinematics is there are positions where there's infinite solutions. If I hold my hand steady or my wrist steady, you could think I can move my shoulder and elbow and my hand will remain steady. So there's a lot of different solutions for the same.
Stefan (33:13):
So there's an optimization problem of where's the best place for my joint to be? Based on some function of different variables and costs.
Ilia (33:20):
And the best part is what you're optimizing for generally for robots is efficiency, either time-wise or energy-wise. And sometimes that could produce undesirable results. Sometimes your robot arm will decide your inverse kinematic math will come out that going from point A to point B requires your arm to swing way around the space to get to point B. And having the arm unpredictably do that in the middle of a trade show would be probably a bad idea.
Stefan (33:43):
You'd have to draw again some faces.
Ilia (33:45):
We'd probably end up hitting some faces. So bounding the box to a small space kind of ensured that the joint solution would always be in the same area.
Stefan (33:54):
So Ilia, I'm watching a video of all these drawings and it's upside down. Why is that?
Ilia (34:00):
That is a bug. Here's the best part, it was a bug that ended up being a feature because I decided it was more kind of mysterious to see it drawing things and at the last second, flip it upside down, it looks sort of more realistic.
Stefan (34:16):
That doesn't look like me at all, oh, I guess.
Ilia (34:18):
It kind of looks like me. It's like those YouTube videos of people drawing upside down, you don't know what they're drawing until the last second. That sort of thing. So it was a bug originally and then I left it in place for fun.
Stefan (34:30):
So another thing that springs to mind with this, you were originally doing that filter layer, just looking at the boundaries. Why wouldn't you look at that, draw a bunch of lines to say, "Here's the face outline, here's the nose outline." Then layer the triangles on top of that because then you could have 50% of triangles, but it would look more face like and less geometric.
Ilia (34:54):
Less geometric. The part you're skipping over there is how do we get from dots in 2D grid, which is your image, to vectors or lines. Because if you try to draw an algorithm that says, "I'm going to cover all these dots with a minimum amount of lines," you're again up to an NP hard problem of what is the minimum amount of lines to connect all of these points? It's the traveling salesman problem all over again. So it's actually a fairly difficult problem to do.
Stefan (35:22):
But if you're just doing the outline of the head, isn't it just, all right, here's kind of half of a circle?
Ilia (35:29):
Yeah, yeah. You could definitely approximate that way. I don't know. I think at the time for trade shows, you're always pretty pressed for time. You're doing this between Europe. So I think I quickly found this SVG library and I was like, "Yeah, good enough." It's more impressionistic this way.
Stefan (35:44):
So if you're to think about doing this again today, and I guess to think about it from a product side, from a business side, I could see myself in the future going to some awful tourist place, some terrible thing, drag your kids and all the children are screaming and all the adults are sad and wishing they remained childless. In places like Disney World, I could imagine there being a robotic arm that draws a picture of a face. I imagine that has some staying power as a tourist thing. So let's say it's no longer 2015 as I've consulted my almanac, it's now 2022. What do you think would be easier about this? How do you think would, and maybe you don't need to be free roaming and drawing people or maybe you can be still. How do you look at this differently?
Ilia (36:35):
Yeah. I don't think in this particular space much has changed unfortunately.
Stefan (36:40):
Really?
Ilia (36:40):
Yeah. I mean you can get slightly different robotic arms. There's more manufacturers of robotic arms, but the algorithmics for inverse kinematics are pretty much the same. The drawing end of it, definitely you could do it better, I threw this together as quickly as possible.
Stefan (36:54):
I guess I'm assuming with stuff like open AI and that great tool that we all use that I can't remember the name of. Not to be an ad for a billion dollar company. I feel like you could come up with a far closer to reality maybe set of strokes. Could you do something more interesting on the ML level to make there be like, here's the individual vectors to draw?
Ilia (37:17):
It might be out there. I haven't seen anybody do research in ML to generate vectors. The trick, here's like a peak behind the curtain of how generating images for dolly and other things work. Basically they start off with a noise image. So they start off with random pixels and then they iterate that through the network that they've trained up until they arrive at the prompt that the person has put in.
Stefan (37:41):
That's something that looks like maybe it could be. I heard that is for example, 18th century oil painting, army of robots, are victoriously battling each other. Which I may or may not have tried all of my dolly granites on.
Ilia (37:57):
Exactly.
Stefan (37:57):
Turns out the thing that was limiting me from being a great artist was not my lack of technical ability. It was in fact my vision and ability to explain it into something that looked like a great piece of art. Even Dolly can't make me a good artist.
Ilia (38:11):
But that whole framework is built on the idea of the information density of a random array of pixels is high enough that you can suss out almost any image you want. And so that's really not working in vectors.
Stefan (38:27):
Basically if you squint at any blurry image, you can come up with any blurry image.
Ilia (38:31):
For the few minute equivalent, yeah.
Stefan (38:32):
And it gets a little less blurry. As a kid, I looked at the tree outside my window, I could see things in the tree imagining, whatever, that type of thing.
Ilia (38:43):
Your childhood sounded interesting, but that whole thing predates on these kind of pixel space. And so I don't know that anybody's tried to do that for vectors. So you might generate a more interesting image. I don't know if you could still generate more accurate vectors, especially to draw a particular person. For dolly or for any of these kind of generative machine learning algorithms, it's pretty hard to draw a particular person. Unless that person is extremely well represented in the training data set. So I could say draw Arnold Schwarzenegger and you get somebody pretty close, but drawing my Uncle Jim-
Stefan (39:24):
He's a great looking guy.
Ilia (39:25):
... going to be very difficult to describe exactly what I want drawn. So yeah, I don't know that ML would help us much.
Stefan (39:32):
But couldn't you just do something like a transform between, here's a captured image of a human, turn this into something that looks like it's drawn on a piece of paper and deduce vectors from that.
Ilia (39:46):
Yeah. You'll still be stuck with the same problem I was stuck with, which again, it's definitely not an unsolvable problem. There's better ways to do than what I threw together, but you could get better line drawings than what I started with. But you still have that translation step of pixel space to vector space, which is challenging even actually if you use open source tools to do that translation software programmatically. If you look really closely, you'll see every single pixel is represented by an individual line, which is definitely not the way you want to do things. That is a bad way to convert from pixel space to vector space. So definitely not unsolvable, but I don't think anything in the last several years has made it drastically easier than it was back then.
Stefan (40:31):
Yeah, that's weird to say. As you think there'd be a row of people out the door thinking, "I want to draw some robots right quick. I want want my robots to draw my Uncle Jim and Arnold Schwarzenegger and of course, an 18th century army depicted in an oil painting, victoriously riding into battle against Napoleon."
Ilia (40:53):
We're going to have to link that image there. So people can see what Stefan's talking about. But yeah, on that note, thank you for chatting about robots today.
Stefan (41:03):
Yeah, absolutely. Thanks for telling me about drawings that require even more imagination than my own to make sense out of. If you are interested in automating things, primarily large things that should not hit other things, you should reach out to us at polymathrobotics.com. Our general autonomy platform makes it really, really easy to automate any large thing so you can focus on the parts that your customers actually care about. Like figuring out if there is in fact a good business in drawing faces as beautiful as mine on a regular basis. Ilia, any other thoughts to close us off with?
Ilia (41:37):
Yeah, be careful with your robots kids.
Stefan (41:41):
Join us again next week. We will draw some more cards. We will talk some more robots. Yeah, I keep on wanting to draw cards at the end of these, and Ilia keeps on shaking his head so vehemently that I think it might fall off.
Ilia (41:54):
It's part of the fun, the unpredictability.
Stefan (41:58):
Just like machine learning.
Ilia (41:59):
Just like machine learning.
Stefan (42:00):
And on that note, thanks for joining us. Talk to you all soon.
Ilia (42:03):
Bye now.
, could you speak or do I just speak for you as CEO?
Ilia (00:23):
Well, I'm just the puppet master behind the strings.
Stefan (00:25):
The strings seem very frayed.
(00:38):
For those of you who are just joining us for the first time at Automate It, we talk about building robots, how we've built robots before. And we start off with a game that I really love where we draw two cards out of a deck, one of which has a technology on it that we either have to fit into a solution or that we might. The other is a setting. And the way to think about this is to imagine two co-founders teaming up, where one of them has some business insight into a particular setting or a market, and the other is a CTO who has some technology they're desperate to use. Or sometimes absolutely would despise and never use. There's no point of it. It's super over-hyped. Ilia, why don't you draw the setting card for us?
Ilia (01:21):
All right. Let's see here, we got senior care.
Stefan (01:26):
Senior care. Man, so as a disclaimer, my wife works in senior care. I have talked to a number of folks about robots in senior care and this will be fun. And we have on the other side, teleoperation.
Ilia (01:40):
The classic mix.
Stefan (01:41):
Yeah.
Ilia (01:43):
Amazing.
Stefan (01:44):
I think there's a couple ways that we could go here. I mean, the most serious could be senior care robots that are teleoperated from some lower labor cost place, that seems like a real business. Or, better than that, I think we could look at every five minutes, we talk about the end of social security in America. And these seniors, they go putzing around working 60, 80 hours for their whole life. They decide at 75 they want to retire and enjoy a fleeting couple of minutes with the grandkids. But they're no longer working and if you're not working, I don't know, what's the point? So why don't we use seniors as teleoperators?
Ilia (02:24):
I was going to say, we should flip the script.
Stefan (02:25):
Yeah.
Ilia (02:26):
I was totally thinking, I wasn't thinking quite so capitalistic hellscape, just thinking more along the lines of maybe vacations for people who have reduced mobility with kind of a VR standpoint.
Stefan (02:39):
Yeah, yeah.
Ilia (02:39):
Let's go with the capitalist hellscape.
Stefan (02:41):
Yeah, let's go the capitalist hellscape. I mean, I think the thing is, so for those of you who don't know, life in a senior care facility is really expensive. So when my wife first moved out here to San Francisco, the senior care facility that she worked in was charging, I want to say eight to $10,000 per bed per month for an assisted living facility. And that's not a place with nurses and doctors. That's a place where there's a nurse's assistant who hands you the medicine and they play old polka on the stereogram.
Ilia (03:11):
That's why cruise ships are so popular. By comparison they seem quite cheap.
Stefan (03:16):
So if you think about it, let's say between your family wealth, your life savings, you have a measly $600,000. That's not going to last you very long if you intend to not croak in the next three to four months. So these seniors, they need to teleoperate. How else are they going to pay their denture bills?
Ilia (03:36):
It's going to be an interesting case because you need a task which requires a human touch, but doesn't require really quick response rates.
Stefan (03:47):
So to be fair, I mean teleop's never really that good in a high response rate. It's like no matter what you're going to see a hundred milliseconds of latency. So yeah, maybe you can drive that up to 20, 25 miles an hour, but it's never really steering the motorcycle around the racetrack at 80 miles an hour. So I feel like actually the seniors aren't going to be that taken away from, that disabled, by having slower reaction time than a 15 year old teleoperator in Paraguay.
Ilia (04:20):
Yeah, that's true. I think the other problem though is the inherent nausea with any kind of VR system or any of that stuff.
Stefan (04:29):
VR systems, honestly, when we did teleop at Starsky, we found that drivers ended up feeling nausea, having trouble feeling a bit disoriented after two, three hours of really high stakes, high speed teleop. So my guess is it's not so much a high speed, you have to be on all the time type of teleop scenario. It might actually be more of maybe autonomy gets you, maybe it's say a robot in a hospital or a robot in a hotel, places where people love the idea of robots. And then you have a senior who's telling people how to get her out of a hotel.
Ilia (05:07):
It could be that. It could be, even in that case, what you're really getting at I think is human supervision of automated tasks. So what you really want is you want kind of human judgment to come in once in a while and most of the time the system is operating itself. But once in a while it gets stuck and you need a human to come in and say, "You know what? Yes, you should do this now because the door is jammed. Go call for help."
Stefan (05:32):
I do feel like we're getting closer and closer into real business. Because now that I think about it, I remember hearing a teleoperated robot where the pitch was seniors. That's deliberately not part of this segment. It's not supposed to be as serious.
Ilia (05:46):
No. Yeah. Let's get back to capitalist hellscape.
Stefan (05:48):
Yeah, capitalist hellscape.
Ilia (05:50):
So being Canadian, American politics are pretty interesting to me.
Stefan (05:56):
Because they're perfect and ideal in every way.
Ilia (05:59):
Absolutely. So I mean with all the voter intimidation that's been happening in some states, we could have predominantly right leaning seniors automate their voter campaign.
Stefan (06:15):
I mean, yeah, we can just take the bomb disposal robots and anywhere where you're say, just to make up a thing that would never happen, anywhere where say it's illegal to hand out water to people waiting in line to vote, you can just have the bomb disposal robot mow people down.
Ilia (06:30):
Actually that's a good point. Is it illegal for people to hand out waters?
Stefan (06:37):
I don't know. I'd have to check the Georgia, I mean Georgia, the country, statutes on who can or cannot hand out water. And I think actually we're opening into an interesting area. Let's say you make it the chat roulette of teleoperation where the teleoperator is paired with a random device for say until they want to pick away. You'd have kind of complete plausible deniability of whether it was your fault or somebody else's that whatever you did. So you could interact with the physical world and then just throw your hands up in the air and blame, I don't know the blockchain that assigns you to that particular robot and note it down for no.
Ilia (07:17):
It's got to be a blockchain. It's always...
Stefan (07:18):
You need the block, this is a great use case for blockchain.
Ilia (07:21):
I heard this really great coin FTX, we should put one hand in for this endeavor. Yeah, no, less accountability for teleop.
Stefan (07:31):
Actually. I mean coming back towards something that might be real but still not, is living vicariously through robots. So not so much like a VR type of headset thing of go see Paris. What I mean is again, bomb disposal robot drives around the elizy whatever, really freaks out a lot of people, including ironically probably some other seniors while all some blue hair in Florida is having the time of her life doing the trip she never got to do.
Ilia (08:02):
Yeah, yeah. And then there there's a full American, the army helps you get where you want to go in life kind of angle that we should think about.
Stefan (08:10):
I think the VA actually. So not a lot of people want to sell weaponized robots to the military. Maybe the way that you breach that subject initially is you can sell non weaponized robots to the VA and give old veterans the ability to travel around the world on treaded vehicles. They're pretty light, they only weigh like one to four tons. For drive around the sidewalk of Europe and Georgia and anywhere else that you're interested in.
Ilia (08:36):
Bring freedom wherever they go. Beer. Yeah. Oh, that got dark. What other teleoperation case could you have?
Stefan (08:47):
So to talk about the serious one, people who work in senior living facilities are actually incredibly hard to staff. This is a hard job that is relatively thankless. Many of the companies that run the senior living facilities are real estate investment trusts with a couple of extra wrinkles around it. So they end up being very bottom line focus. They hire a bunch of blue collar workers. They're paying somewhere between minimum wage and top end retail wages. So let's say in between a McDonald's and a Best Buy. But it's a job that you need special licensing for. When my wife was managing a bunch of people who had this job, many of them would actually have two full-time jobs. They'd be working like 60, 80 hours a week and seeing their senior care job as the way they were giving back, because many of them were from cultures that put a lot of value in taking care of seniors and doing things like that.
(09:45):
So I think if you were willing to make it a cross-border teleop operation system, you really could have people from way Bolivia, Ecuador, Peru, who would just be stoked to give back to an older generation, would be really happy to make the 6 or $7 an hour that system could pay them probably. Because let's say the system's earning 16, $17 an hour, 4 to $6 is profit, the next 6 is for the robot, the next 6 is for the teleoperator. And that's hard. So that adds up to more than $16. But you would have a system like that and end up having not senior care that could solve all problems, but probably you could do some stuff that people can't currently do and the in-person caregivers could spend more of their time focusing on face-to-face connections and adding face-to-face value.
Ilia (10:44):
I mean think the thing you got to think about is how are you defining care, right? Because if you have to do any kind of manipulation task, you immediately blow the cost of the robot. If it's a screen on wheels, it's a few thousand bucks in production.
Stefan (10:58):
A double robot.
Ilia (11:00):
If it's anything that has to pick stuff up, it's immediately a hundred cut.
Stefan (11:04):
Well, I don't think I'd want a robot arm capable of moving engine blocks picking up gram, but I don't know if that is quite interesting because grandma is a little frail and with whatever, 800 pounds per square inch torque, whatever, grandma's not doing so well if there's a bit of controls issues and there's always controls issues.
Ilia (11:27):
No, but I mean even just picking up basic stuff, bring me a glass of water sort of thing. But you could use it as a force multiplier where you still have a few humans around, but you have kind of gopher bots.
Stefan (11:39):
Yeah, I mean you can use them as gofer bots. You could use them as cameras on wheels to check in on people and see if they're doing well. Then as a result you do check-ins on an hour basis, whereas right now it might be three or four hours because a lot of bad things can happen in the extra two to three hours.
Ilia (11:52):
So I feel pretty reasonably people will be uncomfortable with robots checking in on them. I don't know, I'd really love to hear listeners, if anybody out there wants to reach out to us, how has even just telepresence taken off with the senior care kind of case? Because is it very used to, assuming somebody sets it up for you so you don't have to worry about it, it just works.
Stefan (12:17):
I mean, I see ads from Facebook who would never lie to me, that the Facebook telepresence thing always has a picture of a senior talking to their grandkids. And if it's in an ad, it must be true.
Ilia (12:29):
I mean, Facebook only ever lies about election related stuff.
Stefan (12:34):
Ilia getting more than a little political today. I mean to that end, I know many people who in COVID, like seniors very much quickly grabbed onto things like Zoom, things like whatever. In fact, my mom who is now technically a senior, got very into Zoom very quickly and had all the nice things to say about it of, "Now I can talk to this person. Now I can talk to that person."
Ilia (12:58):
So really what you're saying is you want the telepresent system where seniors can visit each other. That sounds like a thing that exists.
Stefan (13:08):
Well, no, more of my point is that with really good robots, and I'm not saying really good like C3PO, I'm saying really good like a double robot with a iPad on a set of wheels that kind of makes it so that you don't have to learn a new UI, actually have to learn how to use a thing. There's just suddenly a face looking at you and there's a camera attached to that face and you can talk to a person. I think that is not too terrifying, not too dystopian and means that that senior is going to see a lot more faces. They're going to have a lot more human interaction. And frankly that if the person on that particular iPad wants to spend in seven minutes talking to the person in that particular room who either has a problem or just wants to say something or whatever, they can do that without being told they need to hit up the next 25 rooms by the end of their shift, which is supposed to be 15 minutes from now.
(14:05):
Because I imagine it doesn't feel nice to be in your golden years, try to tell an interesting story to this nice guy who works in the facility that you live in and him saying, "Yep, yep, yep. That's so great Mr. Stefan, I need to leave because my kids also want to see me. So unless you are having an episode, I'm out of your room now." Whereas if that guy's shift could end after our conversation and someone else could be on another double robot to check it on the other 15 seniors, that might be kind of a best of all worlds.
Ilia (14:40):
And I think the best way to prototype it is with these bomb disposal robots.
Stefan (14:47):
Just batch them. The first folks are going to be chew robot friendly. So you have to make sure that the robot's tough enough to survive.
Ilia (14:55):
That's right. Survival of the robot friendliest.
Stefan (14:59):
I'm just going to say the food at my grandmother's senior living facility was not very good. Think bombs is pretty close to it.
Ilia (15:06):
Oh God.
Stefan (15:08):
With that I think we'll transition to talk to eat some real robots that we worked on. While we've probably alienated all of our listeners over the age of 74, I'm very sorry, Gladys. I'm very sorry, Ethel and Mr. Johnson, I know we're not on the first name basis yet, but I hope you keep our listening to the podcast.
(15:27):
To move on from there Ilia, what we're going to talk today about a robot that you worked on that I always thought was a really interesting idea.
Ilia (15:35):
Yeah. We affectionately named it Fan Gogh after the Fanook carb, was it?
Stefan (15:41):
To zoom out before we get to the naming part, which is the most important part of any robot, I guess to play as similar as we play our currents. What was the point of this overall project? What was the set of technologies you guys were looking at using and then let's kind of work our way to what you guys built.
Ilia (15:58):
Yeah, yeah. So this was a trade show machine, which at Clearpath for trade shows, we've always wanted to have a nice little demo in this particular year. What was time off was-
Stefan (16:08):
To dive into that a little bit, a hard thing about robots as we both know is these things don't work all the time and the most valuable ones look pretty boring after the first five seconds. So building something that is actually demonstrable that fits in a trade show. And also when it comes to what you could do in a trade show, I know how much time you've worked, a lot of shows when you're running around the world in Clearpath, the average group of people who walk up to your booth, how long are they there for?
Ilia (16:36):
You got maybe anywhere between 10 seconds if they're walking by and just want to say hello and up to five minutes if they really want to figure out what's showing.
Stefan (16:44):
So probably a three mile per hour drive around coastal California is off the table.
Ilia (16:50):
Yeah. Probably won't get you the response you're going for.
Stefan (16:55):
Okay. So the problems that then becomes what can you package up that is visually interesting, fits with it... I don't know how big is it 10 feet by 10 feet?
Ilia (17:05):
That's about the scale. You can also get a double 10 by 20, but generally 10 by 10.
Stefan (17:09):
Okay. So yeah, a hundred square feet. You need to catch someone's attention in 10 seconds. Ideally someone who doesn't have a PhD in Robotics, because whatever trade show you go to, there's not many of those.
Ilia (17:24):
I think the Robotics trade shows that we went to by ratio of Robotics species, I think that year the trend had been with the success of Universal Robots, which makes a bunch of human collaborative or cobot arms, a lot of other players were looking to get into human collaborative space. And so at the time-
Stefan (17:44):
Wait what's that mean, human collaborative?
Ilia (17:46):
Robot. Yeah. So that basically means that your robot won't kill you, first of all.
Stefan (17:50):
Well it's totally good. I've heard Asimov not pro robot killing.
Ilia (17:56):
Yeah, definitely the easiest thing to code up is Asimov's laws.
Stefan (18:01):
I just don't know why you can't work on a positronic break. I've been asking this. I've been doing a request for developers. No one seems to have positronic brain experience.
Ilia (18:09):
It's in our backlog.
Stefan (18:11):
Doesn't make sense.
Ilia (18:14):
But more than that, what you want is a system that will interview and that makes a little bit trickier, especially for arms. One is you just get a lot of kinetic energy of theorem spinning around and it's about to hit someone.
Stefan (18:24):
How hard can an arm hit you before it's an ouchie?
Ilia (18:28):
I don't know the exact number, but-
Stefan (18:30):
I was hoping for a three.
Ilia (18:33):
The main problem between a human bumping between you and an arm is that the arm is made out of steel, it tends to move quite a lot faster.
Stefan (18:42):
I don't know many people made a steel.
Ilia (18:44):
Actually side note, but a funny problem with using robot arms for dealing with cell culture is robot arms accelerate so quickly that they kill everything they're trying to move in the little Petri dish. That's really what makes them dangerous as well, is that their acceleration is so high that they get a lot of kinetic energy really quickly in a tiny amount of space. Their hardness combined with their speed, pretty dangerous. But that's only the first step. The second thing is you have pinch points and so you really don't want a finger caught between two joints where it'll get crushed. Because even at low speeds you can easily do that.
Stefan (19:20):
So the trend of the time was human robot collaborative things. You want it to be visually stimulating relatively quickly. It's supposed to be an arm. I guess what are the different ideas? I think of, you always see things like robots playing ping pong. Recently a company was founded out of MongoDB, founder Elliot Horowitz's desire to play chess with a robot arm. What were the early ideas you guys were interested in?
Ilia (19:47):
So in particular CorPath about was working with Fanook who had just released kind of one of their robot arms. They're a big manufacturer in robot arm space primarily. They make arms for industrial use cases. Really heavy, really big-
Stefan (20:00):
Robots [inaudible 00:20:02] and grandma.
Ilia (20:02):
Yeah, exactly. To our earlier discussions, they make really fast stuff. So this is their first, maybe not their first, but one of their early approaches to human collaborative robots. Kind of fun note, the way that they did it is they didn't want to retrofit the arm itself because that'd be expensive and would take a lot of design. So instead of that, they mounted a really accurate force sensor under the arm. And at any given moment as the arm moves, it would predict how much force it should see at the base. And if it saw more or less than that, it knew that it was doing something wrong and it would stop.
Stefan (20:36):
So it's basically a doer, checker architecture for speed of movement?
Ilia (20:40):
Yeah, for inertia, more than speed of movement is for inertia. So that's an interesting case where instead of trying to put force sensors at every joint, we had a big one at the base. It was kind of a clever solution because they don't have to make a new skew. They only had to basically change the paint color from red to green to indicate a cover.
Stefan (21:00):
Green's a great green.
Ilia (21:04):
So we knew we wanted to use that. That was kind of our partner for the year. And the other thing we wanted to use was the Ridgeback base, which at the time was relatively new. Kind of an omni-directional base, kind of stick the two together, what's something interesting we could do. My general manager at the time really wanted to drive around the robot, have it hit symbols to let people know that it was time for the next talk. That seem-
Stefan (21:24):
Driving around the trade show floor.
Ilia (21:28):
But that didn't-
Stefan (21:28):
Driving around the trade show floor with something that looks like a hammer. That's one way to get through [inaudible 00:21:34].
Ilia (21:34):
Very, very safe.
Stefan (21:36):
And so one of the things was hitting symbols. What else was there?
Ilia (21:38):
The other one, which we ended up going with, was actually having it draw on something.
Stefan (21:43):
I mean that thing is pretty easy. I remember watching Bicentennial Man, I recall Robin Williams as a robot drawing things. He was pretty fast at it. How hard can that be? Were you just drawing figure eights or my favorite, the S sign that we all grew up with?
Ilia (21:58):
Yeah. Hey, I started off the project thinking that I could make it draw realistically and that was quickly, classic engineer, [inaudible 00:22:08] is art, whatever. No, very hard. So I really wanted to drive around the conference room, take photos of people and give out a little business card with their drawn kind of photo and our little logo. Start off the process saying, "Okay, let's draw people." And then I realized turning a photo into a series of motion, it's pretty hard.
Stefan (22:31):
Well, yeah, I guess actually I'm interested in that, they have a trial. I guess what tool do you use to even turn that to what this drawing should look like? Let alone the motion planet. Is there a tool that's matrix of 5I face, but turn into shitty line drawing?
Ilia (22:53):
Yeah, I wish there was. I tried a few different approaches. Again, I thought it would be easy, but first approach I did is I took the photo and I did edge detection, standard machine vision tool, kind of look at differences in color-
Stefan (23:09):
What's the end of the person?
Ilia (23:10):
Or where's their features on the face? So any edge, right eyes, edge of eyes, edge of nose, edge of ears. Okay, so great. I have these edges. Now how do I draw them? So now I have a 2D image of pixels of black or white, either edges or no edges, but I still don't know how to turn that-
Stefan (23:29):
These drawings were just a very light outline.
Ilia (23:32):
Well, that's what I thought. That was my first attempt. But then I still realized I haven't actually made my problem that much easier because now I have edges, I have less data, but I don't have data to still turn this into motion. So my second attempt, I luckily stumbled onto a really fun library that would represent an arbitrary photo as a collection of different colored triangles. So it's basically turning it into a mesh and so it can build an image and transfer it into a SVG file, which standard internet file for vector graphics. And the nice thing about vector graphics is it actually defines, it tells you, here's a line and here's how big it is, which is something that's starting to approach motion planning. And so this file would take a photo, turn into triangles. Great. But the triangles, the way that it built the photo was actually filled in colored triangles, which if I told the robot arm to fill in a triangle with a pen would take three hours per photo. So I took that, I made it draw just the outlines and it ended up with a roughly impressionistic photo of the person.
Stefan (24:42):
And say if someone wanted to Google and find these photos and what they look like, what would they search for?
Ilia (24:47):
Yeah. I think it's still on the Clearpath Robotics product page or blog page of Fan Gogh. Sure you could find it. There's a little video of it doing a high speed sketch of me. But yeah, once I landed on this impressionistic style of these triangles, I had my starting and ending point for each triangle. I could translate that 2D plane into motion plans for the robot. And then the Fanook software was good because I could just say, start and end at these locations in 3D space. And it could figure out how to move the arm to do it.
Stefan (25:16):
Well I guess also, what drawing tool was it using?
Ilia (25:19):
Just a pen. There was a pen on the end of the arm and it would tell Fanook apply a certain amount of pressure on this plane and move through-
Stefan (25:27):
I guess how tightly controlled was by this much pressure? Because I feel like the safety things that would stop you from breaking someone's arm are still tolerant of snapping the head off a pen.
Ilia (25:41):
So we cheated a little bit, that's always the rule for good demos is you cheat-
Stefan (25:45):
Or just for robots. Robots is the cheating industry. Don't tell your spouses it's the only way to go.
Ilia (25:54):
So basically what we did is the pen was mounted on a very small spring. And so we always had a little bit of leeway. And so the spring would always apply kind of a constant pressure and the arm could be off by three or four millimeters and the pen would still touch the page.
Stefan (26:09):
I wonder how many people have this picture framed in their house.
Ilia (26:12):
Hopefully everyone at that conference. So I worked, I got it down to about three minutes per person, which was within that kind of attention span.
Stefan (26:23):
And let's talk through that pipeline. So what year roughly was this -ish?
Ilia (26:28):
2015.
Stefan (26:30):
Okay, so 2015. Maybe have a point gray camera or do you have what?
Ilia (26:34):
Cheapo USB webcam.
Stefan (26:36):
Okay. So cheapo USB cam is mounted somewhere just in the booth?
Ilia (26:42):
Nope, right on the robot. The robot would rove and take photos of people.
Stefan (26:45):
It was actually roving taking, that's cool of itself. Okay. What was it following a person with an offset?
Ilia (26:52):
No, it would randomly walk until somebody would see the little placard on it that said, "Press this button to get a sketch." And then people would press the button and that would trigger the operation of take a photo, look for the face using standard machine learning face tools.
Stefan (27:07):
So all this was on a mobile robot, battery powered?
Ilia (27:09):
Yeah.
Stefan (27:10):
Okay.
Ilia (27:10):
So the robot would rob around the conference and it would take photos.
Stefan (27:16):
What's the base of the vehicle again? It's a Clearpath-
Ilia (27:19):
Ridgeback, omnidirectional robot.
Stefan (27:22):
Okay. Well then what were the dimensions of a Ridgeback? How big is this robot?
Ilia (27:28):
Four feet by four feet, probably two feet tall.
Stefan (27:30):
So you wouldn't want to share an elevator with it?
Ilia (27:32):
You could squeeze in.
Stefan (27:34):
Yeah, but it wouldn't be comfortable. It wouldn't be a friendly elevator companion. So it's a four foot by four foot omnidirectional robot. It's got an arm. Does it have anything else? A beret, does it maybe have an accent that makes me feel like it's doing me a favor by drawing me instead of me as the art, the inspiration? Any other pieces of a flair?
Ilia (27:53):
We did 3D print one ear, from the inspiration Van Gogh. So there was a 3D printed ear stuck on the side of it I remember.
Stefan (28:06):
Did that get decided, did people notice that or did that get?
Ilia (28:07):
I don't think people noticed it. I think maybe one or two people got the joke, but it was kind of a small detail.
Stefan (28:13):
It would've been better if after drawing a particularly stressful person it had just ripped one of its ears off and hocked it at the person. As we discussed earlier, a robots thorough things at people are probably-
Ilia (28:24):
Probably a bad idea.
Stefan (28:26):
Yeah. Not a good look. Okay. So this free roving four foot by four foot robot has an arm. It has a cheapo USB camera on it. It's roving around. What speed are we talking? Half a mile an hour, mile an hour?
Ilia (28:38):
Very slow. It has two lidars on the base and it was a very base navigation. Get from point A to point B and very, very slow just because there wasn't really a case for having it drive around quickly.
Stefan (28:51):
And was there a person walking alongside it, or?
Ilia (28:51):
Yes.
Stefan (28:54):
Oh, okay.
Ilia (28:55):
Generally there was a person there to nanny it. Honestly, I don't remember if anybody had to intervene, but I'd be surprised 'cause it was just doing very basic back and forth motion.
Stefan (29:03):
So it had a sign that said, "I'm going to draw a picture of you," and the average person who was probably responding, "Only draw me wearing this diamond necklace."
Ilia (29:13):
Crossover between Bango and Titanic.
Stefan (29:17):
So a user walks up, pushes a button, how'd you target the camera at that point?
Ilia (29:23):
So there's a little display that would show what the robot would see so you had to get in the frame. Yeah, use the humans wherever you can.
Stefan (29:31):
Another cheat code. Why not, just to talk to that decision there, why not just have the camera point itself at the person?
Ilia (29:39):
You could, the fun thing about this one was the algorithm I was running for detecting the face to draw it was a very, very basic face detection algorithm. And in a crowd it would very likely pick out somebody who's not the face that you actually want. So I added on-
Stefan (29:55):
And it's bit more ominous if the robot draws a picture of somebody-
Ilia (29:59):
Somebody else.
Stefan (30:00):
Yeah. Go after this one, yeah.
Ilia (30:03):
Exactly. So very simple heuristic, it would just pick the biggest face it saw on the scene.
Stefan (30:07):
As a person with a very large head, I feel like I would be-
Ilia (30:11):
Personally targeted?
Stefan (30:12):
Yeah. Personally targeted, drawn frequently. I don't know if I feel like I'd prefer that. So you click the button, it targets the biggest head and then it immediately runs the face to triangles?
Ilia (30:27):
Yeah.
Stefan (30:27):
Okay.
Ilia (30:28):
Yeah. So it runs this open source, I forgot the name of it honestly, but it was an open source little SVG generator that then would desaturate, pick the top 30% biggest triangles and then draw out those triangles.
Stefan (30:41):
I mean, did it look much more real with all the triangles or why'd you pick the 30% biggest triangles?
Ilia (30:47):
Because I wanted it to finish in a few minutes. I didn't want it spending hours just-
Stefan (30:52):
Did you see an increase in how good the picture looks at more triangles?
Ilia (30:56):
Oh yeah, for sure. It got more and more realistic. I mean it never got picture perfect. It's kind of an impressionist sort of painting.
Stefan (31:04):
People don't say that we are made of but triangles.
Ilia (31:06):
Yes.
Stefan (31:08):
Okay. So pick the first 30% to stay within that five minute time window. So now you have your drawing. What's the next part of the pipeline?
Ilia (31:15):
So the next part is now have SVG, which actually is a file format that tells me start end points for a bunch of lines. Then I translate that to lines to actually coordinate some 3D space for the arm to move up and down. I forget actually if I mounted the, yeah, I think I mounted the easel it was drawing at, 'cause it was a fixed easel, a little bit on an angle.
Stefan (31:38):
Okay, why?
Ilia (31:40):
Just so it looks nice. And then I just had a little bit of math to translate just up and down, left and right motion to a little bit of depth motion as well.
Stefan (31:48):
What was the full range of motion of the arm? I imagine you just limited it so it wasn't going drawing triangles on people's real faces.
Ilia (31:56):
Yeah, no attacking real humans. We had it had to stay within a roughly one foot by one foot box.
Stefan (32:03):
And I imagine that one foot by one foot box is in the middle of the Clearpath vehicle, meaning that nobody could physically walk within it. Exactly. Also bad to draw on people's legs, they don't care for that.
Ilia (32:15):
Yeah. But I mean the bigger risk is when you're doing arm motion planning that there's this idea of inverse kinetics. So that's basically where, I know where the end of my pen needs to be from my drawing, but I don't know exactly what each joint of my arm needs to do.
Stefan (32:32):
I don't know about you, but I always think about my joints when you're drawing and moving.
Ilia (32:36):
Yeah, exactly. As a human, if you reach for something, you don't think, "I'm going to move my elbow 10 degrees." What you're thinking is, "I'm going to reach this point," and your joints kind of figure it out.
Stefan (32:45):
I mean, that's where the compute is.
Ilia (32:47):
Well that's what the brain's doing. So the equivalent to that for Robotics is inverse kinematics where your endpoint and you kind of solve the problem of where your joints seem to be. The problem is with inverse kinematics is there are positions where there's infinite solutions. If I hold my hand steady or my wrist steady, you could think I can move my shoulder and elbow and my hand will remain steady. So there's a lot of different solutions for the same.
Stefan (33:13):
So there's an optimization problem of where's the best place for my joint to be? Based on some function of different variables and costs.
Ilia (33:20):
And the best part is what you're optimizing for generally for robots is efficiency, either time-wise or energy-wise. And sometimes that could produce undesirable results. Sometimes your robot arm will decide your inverse kinematic math will come out that going from point A to point B requires your arm to swing way around the space to get to point B. And having the arm unpredictably do that in the middle of a trade show would be probably a bad idea.
Stefan (33:43):
You'd have to draw again some faces.
Ilia (33:45):
We'd probably end up hitting some faces. So bounding the box to a small space kind of ensured that the joint solution would always be in the same area.
Stefan (33:54):
So Ilia, I'm watching a video of all these drawings and it's upside down. Why is that?
Ilia (34:00):
That is a bug. Here's the best part, it was a bug that ended up being a feature because I decided it was more kind of mysterious to see it drawing things and at the last second, flip it upside down, it looks sort of more realistic.
Stefan (34:16):
That doesn't look like me at all, oh, I guess.
Ilia (34:18):
It kind of looks like me. It's like those YouTube videos of people drawing upside down, you don't know what they're drawing until the last second. That sort of thing. So it was a bug originally and then I left it in place for fun.
Stefan (34:30):
So another thing that springs to mind with this, you were originally doing that filter layer, just looking at the boundaries. Why wouldn't you look at that, draw a bunch of lines to say, "Here's the face outline, here's the nose outline." Then layer the triangles on top of that because then you could have 50% of triangles, but it would look more face like and less geometric.
Ilia (34:54):
Less geometric. The part you're skipping over there is how do we get from dots in 2D grid, which is your image, to vectors or lines. Because if you try to draw an algorithm that says, "I'm going to cover all these dots with a minimum amount of lines," you're again up to an NP hard problem of what is the minimum amount of lines to connect all of these points? It's the traveling salesman problem all over again. So it's actually a fairly difficult problem to do.
Stefan (35:22):
But if you're just doing the outline of the head, isn't it just, all right, here's kind of half of a circle?
Ilia (35:29):
Yeah, yeah. You could definitely approximate that way. I don't know. I think at the time for trade shows, you're always pretty pressed for time. You're doing this between Europe. So I think I quickly found this SVG library and I was like, "Yeah, good enough." It's more impressionistic this way.
Stefan (35:44):
So if you're to think about doing this again today, and I guess to think about it from a product side, from a business side, I could see myself in the future going to some awful tourist place, some terrible thing, drag your kids and all the children are screaming and all the adults are sad and wishing they remained childless. In places like Disney World, I could imagine there being a robotic arm that draws a picture of a face. I imagine that has some staying power as a tourist thing. So let's say it's no longer 2015 as I've consulted my almanac, it's now 2022. What do you think would be easier about this? How do you think would, and maybe you don't need to be free roaming and drawing people or maybe you can be still. How do you look at this differently?
Ilia (36:35):
Yeah. I don't think in this particular space much has changed unfortunately.
Stefan (36:40):
Really?
Ilia (36:40):
Yeah. I mean you can get slightly different robotic arms. There's more manufacturers of robotic arms, but the algorithmics for inverse kinematics are pretty much the same. The drawing end of it, definitely you could do it better, I threw this together as quickly as possible.
Stefan (36:54):
I guess I'm assuming with stuff like open AI and that great tool that we all use that I can't remember the name of. Not to be an ad for a billion dollar company. I feel like you could come up with a far closer to reality maybe set of strokes. Could you do something more interesting on the ML level to make there be like, here's the individual vectors to draw?
Ilia (37:17):
It might be out there. I haven't seen anybody do research in ML to generate vectors. The trick, here's like a peak behind the curtain of how generating images for dolly and other things work. Basically they start off with a noise image. So they start off with random pixels and then they iterate that through the network that they've trained up until they arrive at the prompt that the person has put in.
Stefan (37:41):
That's something that looks like maybe it could be. I heard that is for example, 18th century oil painting, army of robots, are victoriously battling each other. Which I may or may not have tried all of my dolly granites on.
Ilia (37:57):
Exactly.
Stefan (37:57):
Turns out the thing that was limiting me from being a great artist was not my lack of technical ability. It was in fact my vision and ability to explain it into something that looked like a great piece of art. Even Dolly can't make me a good artist.
Ilia (38:11):
But that whole framework is built on the idea of the information density of a random array of pixels is high enough that you can suss out almost any image you want. And so that's really not working in vectors.
Stefan (38:27):
Basically if you squint at any blurry image, you can come up with any blurry image.
Ilia (38:31):
For the few minute equivalent, yeah.
Stefan (38:32):
And it gets a little less blurry. As a kid, I looked at the tree outside my window, I could see things in the tree imagining, whatever, that type of thing.
Ilia (38:43):
Your childhood sounded interesting, but that whole thing predates on these kind of pixel space. And so I don't know that anybody's tried to do that for vectors. So you might generate a more interesting image. I don't know if you could still generate more accurate vectors, especially to draw a particular person. For dolly or for any of these kind of generative machine learning algorithms, it's pretty hard to draw a particular person. Unless that person is extremely well represented in the training data set. So I could say draw Arnold Schwarzenegger and you get somebody pretty close, but drawing my Uncle Jim-
Stefan (39:24):
He's a great looking guy.
Ilia (39:25):
... going to be very difficult to describe exactly what I want drawn. So yeah, I don't know that ML would help us much.
Stefan (39:32):
But couldn't you just do something like a transform between, here's a captured image of a human, turn this into something that looks like it's drawn on a piece of paper and deduce vectors from that.
Ilia (39:46):
Yeah. You'll still be stuck with the same problem I was stuck with, which again, it's definitely not an unsolvable problem. There's better ways to do than what I threw together, but you could get better line drawings than what I started with. But you still have that translation step of pixel space to vector space, which is challenging even actually if you use open source tools to do that translation software programmatically. If you look really closely, you'll see every single pixel is represented by an individual line, which is definitely not the way you want to do things. That is a bad way to convert from pixel space to vector space. So definitely not unsolvable, but I don't think anything in the last several years has made it drastically easier than it was back then.
Stefan (40:31):
Yeah, that's weird to say. As you think there'd be a row of people out the door thinking, "I want to draw some robots right quick. I want want my robots to draw my Uncle Jim and Arnold Schwarzenegger and of course, an 18th century army depicted in an oil painting, victoriously riding into battle against Napoleon."
Ilia (40:53):
We're going to have to link that image there. So people can see what Stefan's talking about. But yeah, on that note, thank you for chatting about robots today.
Stefan (41:03):
Yeah, absolutely. Thanks for telling me about drawings that require even more imagination than my own to make sense out of. If you are interested in automating things, primarily large things that should not hit other things, you should reach out to us at polymathrobotics.com. Our general autonomy platform makes it really, really easy to automate any large thing so you can focus on the parts that your customers actually care about. Like figuring out if there is in fact a good business in drawing faces as beautiful as mine on a regular basis. Ilia, any other thoughts to close us off with?
Ilia (41:37):
Yeah, be careful with your robots kids.
Stefan (41:41):
Join us again next week. We will draw some more cards. We will talk some more robots. Yeah, I keep on wanting to draw cards at the end of these, and Ilia keeps on shaking his head so vehemently that I think it might fall off.
Ilia (41:54):
It's part of the fun, the unpredictability.
Stefan (41:58):
Just like machine learning.
Ilia (41:59):
Just like machine learning.
Stefan (42:00):
And on that note, thanks for joining us. Talk to you all soon.
Ilia (42:03):
Bye now.
Want to stay in the loop?
Get updates & robotics insights from Polymath when you sign up for emails.
