Lego 51515 Bluetooth Object Detection Robot

[osullivp]

Hi all,

I've been working on a Lego 51515 robot that uses computer vision to detect an object and then pick it up. Here is a video:

The robot works as follows:

1) Objects are detected by a free TensorFlow Android App, available on Google Play (ArdObjectTracker)

2) The detection data is sent from the mobile via Bluetooth Classic and picked up on the Arduino board by an HC-05 module

3) The object detection data is then processed and converted to robot movement instructions which are then sent to the Lego hub via the BluetoothLE module (HM-10)

4) Finally, the Lego hub (running a Python program) receives the instructions via a BluetoothLE notification and then moves to pick up the object

I have shared the code and more details on this Github page:

https://github.com/osullivp/Lego51515BLEArduinoTensorFlow

Hope you all find this helpful!

Regards,

Paul.

 

 

Edited February 26, 2021 by osullivp

[Gimmick]

I realy like it! Great work!

[osullivp]

7 hours ago, Gimmick said:

I realy like it! Great work!

Thanks!

I wanted to demonstrate an object detection capability without the need for TensorFlow programming.

I also wanted to demonstrate connecting to an Arduino board using the ubluetooth python module on the Lego Hub.

I've shared the full code for both the Lego Hub and the Arduino elements in the Github repo here:

https://github.com/osullivp/Lego51515BLEArduinoTensorFlow

Any questions, please let me know 

Edited February 26, 2021 by osullivp

[BrickDesignerNL]

Why is the arduino needed in between? The Hub has BT as well, right? So can't we just let the hub receive the data directly from the Android app via BT and process it in python on the Hub?

[osullivp]

Hi @BrickDesignerNL,

The reason for the Arduino board is that the particular Android app I am using transmits data via Bluetooth Standard whereas the Lego Hub uses BluetoothLE via Micropython. Possible alternatives would be:

• Find a TensorFlow Android app on Google Play or elsewhere that transmits object detection data via BluetoothLE. I wasn't able to find one but if one exists that would simplify things
• Code a TensorFlow Android app myself that implements the TensorFlow API and transmits object detection data via BluetoothLE. I could do this but it would be more time consuming

Regards,

Paul.

Edited May 6, 2021 by osullivp

[ninoguba]

On 5/6/2021 at 9:15 AM, osullivp said:

Hi @BrickDesignerNL,

The reason for the Arduino board is that the particular Android app I am using transmits data via Bluetooth Standard whereas the Lego Hub uses BluetoothLE via Micropython. Possible alternatives would be:

• Find a TensorFlow Android app on Google Play or elsewhere that transmits object detection data via BluetoothLE. I wasn't able to find one but if one exists that would simplify things
• Code a TensorFlow Android app myself that implements the TensorFlow API and transmits object detection data via BluetoothLE. I could do this but it would be more time consuming

Regards,

Paul.

There is a post by Nard on the Spike/Mindstorms FB groups that does exactly that.   You may want to check it out.

[osullivp]

On 5/8/2021 at 7:08 PM, ninoguba said:

There is a post by Nard on the Spike/Mindstorms FB groups that does exactly that.   You may want to check it out.

Hi @ninoguba

Thanks for this. For everyone's reference, Nard's post can be found here:

https://www.facebook.com/groups/SPIKEcommunity/permalink/1357979457913616/

In the post there is a demo of a TensorFlow Android app talking directly to the Spike Prime hub (presumably via Bluetooth LE). The demo shows a Spike Prime robot tracking a cup as the cup is moved around. It's not clear if it's an off the shelf app or developed by Nard as there are currently no details in the post. I would ask myself but I don't do Facebook :-)

Hope this post helps other people looking for a TensorFlow solution that can be used with Spike Prime/51515.

Regards,

Paul.

[Breno Thiago]

Hello friend, how are you? I need to make an integration between arduino and lego robot inventor hub, could you help me with that? Is there any way I can get in touch with you?

Edited June 8, 2023 by Breno Thiago