Bad apple on esp32
Recently i've got my hands on a esp32 and a small display for it, this article is a story, and some info, on how i got it running.
Hardware/software used:
- 2.8inch Touch LCD Shield Rev 2.1 (by WaveShare) (240x320) (st7789)
The main goal of the project for me, was to learn some C. I've never got to use it in a real project, and this is a good opportunity. Because I'm new to it, my code may be of questionable quality. I decided to not use arduino ide & its libraries, since they seemed too abstract for my learning goal.
Skill issues
The first obstacle was getting the SPI to work. Somehow i read the gpio numbers wrong, meaning that each of my connectors was attached to 1 gpio lower then intended. It took me messing around with osciloscope to figure out my mistake.
After this i was able to make a fun cool color show using the esp.
At first i tried messing around with the SPI api directly but was really unsuccessfull. It turned out that esp idf has a nice api for different kinds of spi controlled displays.
Sending Images
To mess around with anything more serious I need a frame buffer - a place to store the pixel data, and for 16 bit colors (2 bytes) x 320(height) x 240 (width) thats 150KB. My esp has 512KB of SRAM, so I'm using ~30% just to hold pixel data, this could be much tighter on other chips.
For a test I simply allocated the required memory and displayed it.
This might look weird but its expected. The display shows whatever was in the allocated buffer. Since this will be overwritten by an image there is no reason to clear it beforehand.
To transfer the image, i decided to connect the esp to WiFi and transfer data over TCP. I tried using udp, but for some reason I had massive packet loss, either the esp was not keeping up, or my WLAN is messed up. Either way switching to tpc was simpler than troubleshooting udp.
For the program that sends the image over the network, i decided to use go (programming language). This way i can easily pre process the image on my pc before sending it over.
The result:
Sooo... to transfer video we just send multiple images... YES... and NO, sadly when sending multiple images the esp couldn't keep up. By just spamming images i was able to get 2-5 fps and the chip started overheating. I could've tried compression/sending only the differences, which in theory should help, but I wasn't sure if forcing the esp to do more work was a good idea. Soo.. I decided to fucus on my final goal.
Streaming Bad Apple
Bad Apple is grayscale, but mostly just black an white. The only parts where there is any "gray" are edges and some effects, but making the image just black and white doesn't impact the experience that much. This way we can have 8 pixels/byte (1 bit color space) instead of the previous 0.5 pixels/byte. This reduces the sent image size from 153600 bytes to 9600, that's a 93.75% decrease in size.
The second problem is that go doesn't seem to have a good video library. But since i already have code to transfer images, why not use that. I just used:
simsin@arch:~$ ffmpeg -i bad_apple.webm -pix_fmt rgba frames/%04d.png
This command converted the video into over 6500 frame pngs. This way i can use the old code with minimal modifications, to get this result:
The final project can be found on my github.