Together with friends, I made a Pong game that you can play with your body (videos below). This is how it happened.
At my university, we do a semester project for our computer science bachelor. Our project was to create and test an emergency brake mechanism for an RC Car. For detecting whether the car would run over a pedestrian, we had the opportunity to use a LiDAR: a super expensive laser scanner with very good precision. The LiDAR was mounted on the car, together with an ODROID minicomputer it was connected to. The ODROID ran Ubuntu and we used ROS (the Robot Operating System) for controlling the AC car.
We definitely wanted a visualization of the live LiDAR data to be able to understand it better. This data was just a cloud of points—with information about how well the LiDAR recognized each point. I wrote some software that livestreams this data to a website via WebRTC to visualize it. WebRTC (Web Real-Time Communication) is a lossy UDP-like network protocol. If you prefer data packets arriving fast over each single packet arriving and doing so in the right order, WebRTC is you protocol of choice. Usually, it is used to transmit webcam data, but it can transmit arbitrary data, like coordinates of points in our case.
To get the LiDAR data to the browser, I first used a named pipe, also called FIFO. It is basically just a file you can write in. The ROS process continously writes to the FIFO and a Node.js process reads from it. Then, the Node.js process communicates it to the browser via WebRTC. It first uses the library socket.io to do a handshake with the browser and exchange information necessary for connecting via WebRTC. Then, simple-peer, another JavaScript library, handles the WebRTC connection. I was surprised by how complicated it is to actually use WebRTC. Now, I understand why many online services and games (like agar.io, for example) just use WebSockets, which use the slower, lossless, and packet-order-guaranteeing TCP under the hood instead of UDP.
Once all this was done, we could implement a simple cluster detection algorithm to analyze the point cloud and recognize something being in the RC Car's way. Then we connected all parts together to make the car actually brake, and tested everything rigorously.
Playing Pong with your body
After the project was done, a friend of mine and I decided to have some fun with the tech. We built a pong game that you can steer with your body. This is how it works: When the cluster detection algorithm detects a cluster in a certain coordinate range (a box), it maps the center of the cluster to the center of a pong bat. The games runs in the browser in HTML5 canvas through JavaScript. I made a basic game engine with box collision detection for the pong game that is kinda similar to P5.js. I wrote it so that it would easily generalize to higher dimensions but I didn't test that out. This is what it all looks like.
This was a super fun project. Thanks to WebRTC, the game was actually quite responsive and playable. Relearning how to make a basic game engine was interesting. We also made the ball say ouch every time it hit something. Good times.
Links: