I’m back from taking a break. I’ve been working on a number of projects that culminated a crunch-time scenario where there was lots to get done before I went on a vacation for which I would have no computer and an Internet connection that would range from flaky-to-none. When going on trips I often bring some movies with me to watch when I’m taking a break and retreating to my lodging. For this trip, I did something a bit different than usual. Instead of copying my media directly to my mobile device, I decided I would copy it to a Raspberry Pi and set it up as a movie streaming server. (More on elements of my particular scenario at the end of this post) Early last year I had posted something on using Express and NodeJS for streaming. For this trip, I copied that to a Pi that had a lot of storage and did some other things to prepare it for my environment. These included:
- Setting up the Pi as a Wireless Access Point
- Setting my Application to Run on Boot
- Adding the Ability to Upload files
While I didn’t perform these changes during my trip, I realized there was some other functionality I wanted this solution to have, including
- Converting videos of other formats to MP4
- Ensuring the MP4’s metadata was arranged for streaming
- Adding some service to convert uploaded files as needed
Many of these changes could be topics of their own. Let’s talk about uploading files.
Uploading a File with express-fileupload
My first solution for uploading a file was to make a simple web form allowing a file to be selected and add a router to my web application to handle file uploads. I’ll start off telling you that this didn’t work. Skip ahead to the next section if you don’t want to read about this initial failed attempt. The webform itself is simple. A minimalist version of it follows.
<form action='/upload' enctype='multipart/form-data' method='post'>
<input type='file' id='videoUploadElement' name='video' />
<input type='submit' />
</form>
On the server-side, I added the a router to accept my file uploads and write them to a target folder. Upon receiving the bits of the file, I’m writing it to a designated folder preserving the original file name.
//Do not use. For demonstration purposes only. While this
//appears to work. But the entire file is loaded to memory
//before being written to disk. I've found with large files
//this can easily slow down or halt the Pi. Use the router
const express = require('express');
const router = express.Router();
router.post('/', async(req,res) => {
try {
if(!req.files) {
console.log(req.body);
res.send({
status: false,
message: 'No File'
});
} else {
let video = req.files.video;
video.mv('./uploads/' + video.name);
res.send({
status: true,
message: ' File Uploaded',
data: {
name: video.name,
mimetype: video.mimetype,
size: video.size
}
});
}
} catch(err) {
res.status(500).send(err);
}
});
module.exports = router;
Testing it out on a small file, it works just fine. Testing it out on a large file locks up the Pi. The issue here is that **all** of the file must be loaded to memory before it is written to storage. The files I was uploaded were feature-length high-definition videos. I tried uploading a 4.3 gig video and saw the Pi progress from less responsive to all responsive. This wasn’t going to work for my needs.
Uploading and Streaming to Storage with Busboy
Instead of using express-fileupload I used connect-busboy. Using Busboy I’m able to stream the bits of the file to storage as they are being uploaded. The complete file does **not** need to be loaded to memory. When the upload form sends a file, the busboy middleware makes the file’s data and meta-data available through the request object. To save the file, create a file stream to which the data will be written and pipe the upload (busboy) to the file storage stream.
//Uses Busboy. See: https://github.com/mscdex/busboy
const express = require('express');
const router = express.Router();
const path = require('node:path');
const Busboy = require('connect-busboy');
const fs = require("fs");
const UPLOAD_FOLDER = 'uploads';
if(!fs.existsSync(UPLOAD_FOLDER)) {
fs.mkdirSync(UPLOAD_FOLDER, { recursive: true });
}
router.post('/', async(req,res, next) => {
try {
console.log('starting upload');
console.log(req.busboy);
req.busboy.on('field', (name, val, info)=> {
console.log(name);
});
req.busboy.on('file', (fieldname, uploadingFile, fileInfo) => {
console.log(`Saving ${fileInfo.filename}`);
var targetPath = path.join(UPLOAD_FOLDER, fileInfo.filename);
const fileStream = fs.createWriteStream(targetPath);
uploadingFile.pipe(fileStream);
fileStream.on('close', ()=> {
console.log(`Completed upload ${fileInfo.filename}`);
res.redirect('back');
});
});
req.pipe(req.busboy);
} catch (err) {
res.status(500).send(err);
}
});
module.exports = router;
A fairly complete version of my app.js follows, including the reference to the router that has the streaming upload method.
const express = require('express');
const createError = require('http-errors');
const path = require('path');
const { v4: uuidv4 } = require('uuid');
const Busboy = require('connect-busboy');
require('dotenv').config();
app = express();
app.use(Busboy({
immediate: true,
limits: 10* 1073741824,
highWaterMark: 4 * 1048576, // Set 4 megabyte buffer
}));
app.use(express.static('public'));
const libraryRouter = require('./routers/libraryRouter');
const videoRouter = require('./routers/videoRouter');
const uploadRouter = require('./routers/uploadRouter');
const streamRouter = require('./routers/streamRouter');
app.use('/library', libraryRouter);
app.use('/video', videoRouter);
app.use('/upload', uploadRouter);
app.use('/streamUpload', streamRouter);
app.use(function (req, res, next) {
console.log(req.originalUrl);
next(createError(404));
});
app.set('views', path.join(__dirname, 'views'));
app.engine('html', require('ejs').renderFile);
app.set('view engine', 'html');
With this in place, I have a working upload feature. I don’t want to stop there though. FFMPEG runs on the Pi and I may take advantage of this to do additional transformations on media to prepare it for streaming. I’m going to add code for handling this processing next.
GEeting the Code
If you want to try the code out, it is on GitHub. You can find it here: https://github.com/j2inet/VideoStreamNode.
About my Scenario
I only brought the Pi for movies, but I ended up using it for a lot more. I had an iPhone and an Android phone with me on the trip. I managed to fill up the memory on my iPhone and needed a place to move some video. The Android phone had plenty of space, but there was no direct method to transfer data between the two devices. At one point I also wanted to edit video using Adobe Rush, only to find that Adobe doesn’t allow Rush to be installed on Samsung’s latest generation of phones at this time. My iPhone had Rush, thus I had to get information transferred from my Android Phone to my iPhone to do this.
Had these use cases come to mine I probably would have acquired a USB adapter for my Phone (the Android phone will connect to my USB-C drives with no special adapters needed). I’m sure there are apps that could be used to bridge this gap, but my iPhone has no coverage when I’m out of the country. I didn’t have access to try out such apps. Some part of me feels that it is a bit silly that data can’t move between these two systems without some intermediary, such as another computer, or dongle and drives. But given that we are still not at a point where media in text messages is reliably transferred without quality loss it isn’t surprising.
Given more time, I might make an app specifically for transferring data between devices in the absence of a common network. (This is not something I am promising to do though).
Posts may contain products with affiliate links. When you make purchases using these links, we receive a small commission at no extra cost to you. Thank you for your support.
Twitter: @j2inet
Instagram: @j2inet
Facebook: j2inet
YouTube: j2inet
Telegram: j2inet
j2i.net 