Month: November 2022

Auto-Syncing Node on Brightsign for Development

I’m working on a Node project that runs on the BrightSign hardware. To run a node project, one only needs to copy the project files to the root of a memory card and insert it into the device. When the device is rebooted (from power cycling) the updated project will run. For me, this deployment process, though simple, has a problem when I’m in the development phase. There’s no way for me to remotely debug what the application is doing, this results in a lot more trial-and-error to get something working correctly since there’s no access to the information of some errors. Usually when testing the effect of some code change I’m developing directly on a compatible system and can just press a run project see it execute. Copying the project to a memory card, walking over to the BrightSign, inserting the card, removing the power connector, and then reinserting the power connector takes longer. Consider that there are many cycles of this that must be done and you can see there is a productivity penalty.

In seeking a better way, I found an interface in the BrightSign diagnostic console that allows someone to update individual files. Navigating to the BrightSign’s IP address displays this console.

It doesn’t allow the creation of folders, though. After using the Chrome diagnostic console to figure out what endpoints were being hit I had enough information to update files through my own code instead of using the HTML diagnostic interface. Using this information, I was able to make a program that would copy files from my computer to the BrightSign over the network. This functionality would save me some back-and-forth in moving the memory card. I still needed to perform the initial copy manually so that the necessary subfolders were in place. There’s a limit of 10 MB per file when transferring files this way. This means that media would have to be transferred manually. But after that, I could copy the files without getting up. This reduces the effort to running new code to power cycling the device. I created a quick .Net 7 core program. I used .Net core so that I could run this on my PC or my Mac.

The first thing that my program needs to do is collect a list of the files to be transferred. The program accepts a folder path that represents the contents of the root of the memory card and builds a list of the files within it. 

public FileInfo[] BuildFileList()
{
    Stack<DirectoryInfo> directoryList = new Stack<DirectoryInfo>();
    List<FileInfo> fileList = new List<FileInfo>();
    directoryList.Push(new DirectoryInfo(SourceFolder));

    while(directoryList.Count > 0)
    {
        var currentDirectory = new DirectoryInfo(directoryList.Pop().FullName);
        var directoryFileList = currentDirectory.GetFiles();
        foreach(var d in directoryFileList)
        {
            if(!d.Name.StartsWith("."))
            {
                fileList.Add(d);
            }
        }                
        var subdirectoryList = currentDirectory.GetDirectories();
        foreach(var d in subdirectoryList)
        {
            if(!d.Name.StartsWith("."))
            {
                directoryList.Push(d);
            }
        }
    }
    return fileList.ToArray() ;
}

To copy the files to the machine, I iterate through this array and upload each file, one at a time. The upload requests must include the file’s data and the path to the folder in which to put it. The root of the SD card is in the file path sd, thus all paths will be prepended with sd/. To build the remote folder path, I take the absolute path to the source file, strip off of front part of that path and replace it with sd/. Since I only need the folder path, I also strip the file name from the end.

public void UploadFile(FileInfo fileInfo)
{
    var remotePath = fileInfo.FullName.Substring(this.SourceFolder.Length);
    var separatorIndex = remotePath.LastIndexOf(Path.DirectorySeparatorChar);
    var folderPath= "sd/"+remotePath.Substring(0, separatorIndex);
    var filePath = remotePath.Substring(separatorIndex + 1);
    UploadFile(fileInfo, folderPath);
}

With the file parts separated, I can perform the actual file upload. It is possible the wrong path separator is being used since this can run on a Windows or *nix machine. I replace instances of the backslash with the forward slash. The exact remote endpoint to be used has changed with the BrightSign firmware version. I am using a November 2022. For these devices, the endpoint to write to is /api/v1/files/ followed by the remote path. On some older firmware versions, the path is /uploads.html?rp= followed by the remote folder path.

public void UploadFile(FileInfo fileInfo, string remoteFolder)
{
    if (!fileInfo.Exists)
    {
        return;
    }
    remoteFolder = remoteFolder.Replace("\\", "/");
    if (remoteFolder.EndsWith("/"))
    {
        remoteFolder = remoteFolder.Substring(0, remoteFolder.Length - 1);
    }
    if (remoteFolder.StartsWith("/"))
    {
        remoteFolder = remoteFolder.Substring(1);
    }

    String targetUrl = $"http://{BrightsignAddress}/api/v1/files/{remoteFolder}";
    var client = new RestClient(targetUrl);
    var request = new RestRequest();
    request.AddFile("datafile[]", fileInfo.FullName);
    try
    {
        var response = client.ExecutePut(request);
        Console.WriteLine($"Uploaded File:{fileInfo.FullName}");
        //Console.WriteLine(response.Content);
    }
    catch (Exception ect)
    {
        Console.WriteLine(ect.Message);
    }
}

I found if I tried to upload a file that already existed, the upload would fail. To resolve this problem I made a request to delete a file before uploading it. If the file doesn’t exists when the delete request is made, no harm is done. 

        public void DeleteFile(string filePath)
        {
            filePath = filePath.Replace("\\", "/");
            if(filePath.StartsWith("/"))
            {
                filePath = filePath.Substring(1);
            }            
            string targetPath = $"http://{this.BrightsignAddress}/delete?filename={filePath}&delete=Delete";
            var client = new RestClient(targetPath);
            var request = new RestRequest();            
            try
            {
                var response = client.ExecuteGet(request);
                Console.WriteLine($"Deleted File:{filePath}");                
            }
            catch (Exception ect)
            {
                Console.WriteLine(ect.Message);
            }
        }

When I had the code this far, I was saved some time. To save even more I used the FileSystemWatcher to trigger my code when there was a change to any of the files.

          FileSystemWatcher fsw = new FileSystemWatcher(@"d:\\MyFilePath");
            fsw.NotifyFilter = NotifyFilters.CreationTime | NotifyFilters.FileName | NotifyFilters.Size | NotifyFilters.LastWrite;
            fsw.Changed += OnChanged;
            fsw.Created+= OnChanged;
            fsw.IncludeSubdirectories= true;
            fsw.EnableRaisingEvents= true;

My change handler uploads the specific file changed. It was still necessary to power-cycle the machine. But the diagnostic interface also had a button for rebooting the machine. With a little bit of probing, I found the endpoint for requesting a reboot. Instead of rebooting every time a file was updated, I decided to reboot several seconds after a file was updated. If another file is updated before this several seconds has passed, then I delay for more time. This way if there are several files being updated there is a chance for the update operation to complete before a reboot occurs. 

static System.Timers.Timer rebootTimer = new System.Timers.Timer(5000);

public void Reboot()
{
    string targetPath = $"http://{this.BrightsignAddress}/api/v1/control/reboot";
    var client = new RestClient(targetPath);
    var request = new RestRequest();
    var response = client.ExecutePut(request);
}

static void OnChanged(object sender, FileSystemEventArgs e)
{
    var f = new FileInfo(e.FullPath);
    s.DeleteFile(f);
    s.UploadFile(f);
    rebootTimer.Stop();
    rebootTimer.Start();
}

Now, as I edit my code the BrightSign is being updated. When I’m ready to see something run, I only need to wait for a reboot. The specific device I’m using takes 90 seconds to reboot. But during that time I’m able to remain being productive.

There is still room for improvement, such as through doing a more complete sync and removing files found on the remote BrightSign that are not present in the local folder. But this was something that I quickly put together to save some time. It made more sense to have an incomplete but adequate solution to save time. Creating this solution was only done to save time on my primary tasks. Were I to spend too much time with it then it is no longer a time saver, but a productivity distraction.

Posts may contain products with affiliate links. When you make purchases using these links, we receive a small commission at no extra cost to you. Thank you for your support.

Mastodon: @j2inet@masto.ai
Instagram: @j2inet
Facebook: @j2inet
YouTube: @j2inet
Telegram: j2inet
Twitter: @j2inet

SD Pro Plus Micro SD XC memory card

Avoiding Unnecessary File Downloads While Syncing

I had the opportunity to revisit an old project that was created for a client. The initial release of this project had a program that was syncing content from a CMS. It was made to only download content that had been downloaded since the last time it synced. For some reason, it was now always downloading all of the files instead of only the ones that change. Looking into the problem I found that changes in the CMS resulted in files no longer having ETAG headers, which are used to tell if a file has changed since the last time it was requested. The files still had a header indicating a last updated date. It is easy enough to use that header instead. But the client had enough requests for changes to justify writing a new syncing component; they had a new CMS with different APIs. File syncing isn’t complex, I could rewrite the component easily in an evening. I decided to write the new version of the component using .Net 6.0.

Before downloading a file, I need to check the attributes of the file on the server end without starting the transfer of the file itself. The HTTP verb for obtaining this information is HEAD. The HEAD verb will return the headers for the resource identified by the URI, but it doesn’t return the resources data stream itself. As a quick test, I grabbed the URL for an MP3 player I keep seeing in an Amazon advertisement. https://m.media-amazon.com/images/I/61TUVbqPhLL.AC_SL1500.jpg.

The MP3 Player Image from the Amazon Advertisement

I used Postman to request the image at the URL and examined the headers. Postman will perform a GET request by default. Changing the request from GET to HEAD results in a response with no body, but has headers. This is exactly what we want!

There are a couple of things that we will need to do with this information. We will need to save it somewhere for future use. When we make future requests, we need to use this information to filter what data we transfer. The filtering can be done on the client side within the logic of the program making the request, or it can be performed on the server side by adding an additional header to the request named If-Modified-Since. Providing a date in this header will cause the server to either send the new resource (if it is more recent than the date in this parameter) or it will return header information only (if the server version is not more recent than the date specified). The date must be in a specific format. But if you are saving the original date response, then you can use it as it was received.

Let’s jump into actual code. I’ve made a data class that stores information about the files I will be downloaded. 

namespace FileSyncExample.ViewModels
{
    public class FileData: ViewModelBase
    {
        private DateTimeOffset? _serverLastModifiedDate;
        [JsonProperty("last-modified")]
        public DateTimeOffset? ServerLastModifiedDate
        {
            get => _serverLastModifiedDate;
            set => SetValueIfChanged(() => ServerLastModifiedDate, () => _serverLastModifiedDate, value);
        }

        public string _fileName;
        [JsonProperty("file-name")]
        public string FileName
        {
            get => _fileName;
            set => SetValueIfChanged(() => FileName, () => _fileName, value);
        }

        private string _clientName;
        [JsonProperty("client-name")]
        public string ClientName
        {
            get => _clientName;
            set => SetValueIfChanged(()=>ClientName, () => _clientName, value);
        }

        private bool _didUpdate;
        [JsonIgnore]
        public bool DidUpdate
        {
            get => _didUpdate;
            set => SetValueIfChanged(()=>DidUpdate, ()=>_didUpdate, value);
        }
    }
}

I’m using this for two purposes in this example program. I’m both building a download list with it and am using it to save metadata. In the real program, this list is made using a query to the CMS. I create a list of these objects with the identifiers.

        public MainViewModel()
        {
            Files.Add(new FileData() { FileName= "61lLJ85GYXL._AC_SL1000_.jpg" });
            Files.Add(new FileData() { FileName= "61qfFAQ3xKL._AC_SL1500_.jpg" });
            Files.Add(new FileData() { FileName= "71PKvcmV6DL._AC_SX679_.jpg" });
            Files.Add(new FileData() { FileName= "71fOsWX9qlL._AC_UY327_FMwebp_QL65_.jpg" });
        }

All of these images are coming from Amazon. The full URL to the data stream is built by prepending the file name. I do this through a string format.

var requestUrl = $"https://m.media-amazon.com/images/I/{file. Filename}";

For the download, I am using the HttpClient. It accepts a request and returns the response. 

HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.ConnectionClose = true;

For now, let’s code for a single scenario; there are no files already downloaded. We wish to do our priming download and save the file’s data and the metadata about the file. To keep the file system clean instead of placing the metadata in a separate file I’m saving it in an alternative data stream. This only works on NTFS file systems. If you would like to learn more about that read here. The significant parts of the code to perform the download follows.

var requestUrl = $"https://m.media-amazon.com/images/I/{file.FileName}";
var request = new HttpRequestMessage(HttpMethod.Get, requestUrl);
var response = await client.SendAsync(request);
var lastModified = response.Content.Headers.LastModified;
if(lastModified.HasValue)
{
    file.ServerLastModifiedDate = lastModified;
}
try
{
    response.EnsureSuccessStatusCode();
    using (FileStream outputStream = new FileStream(Path.Combine(Settings.Default.CachePath, file.FileName), FileMode.Create, FileAccess.Write))
    {
        var data = await response.Content.ReadAsByteArrayAsync();
        outputStream.Write(data, 0, data.Length);
    }
    //Putting the metadata in an alternative stream named meta.json
    var fileMetadata = JsonConvert.SerializeObject(file);
    Debug.WriteLine(fileMetadata);
    var metaFilePath = Path.Combine(Settings.Default.CachePath, $"{file.FileName}:meta.json");
    var fileHandle = NativeMethods.CreateFileW(metaFilePath, NativeConstants.GENERIC_WRITE,
                        0,//NativeConstants.FILE_SHARE_WRITE,
                        IntPtr.Zero,
                        NativeConstants.OPEN_ALWAYS,
                        0,
                        IntPtr.Zero);
    if(fileHandle != IntPtr.MinValue)
    {
        using(StreamWriter sw = new StreamWriter(new FileStream(fileHandle, FileAccess.Write)))
        {
            sw.Write(fileMetadata);
        }
    }

}
catch(Exception exc)
{

}

After running the program, the images show in my download folder. When I open PowerShell and check the streams, I see my alternative data stream present.

Printing out the data in one of the alternative data streams, I see the data in the format that I expect. 

PS C:\temp\streams> Get-Item .\61lLJ85GYXL._AC_SL1000_.jpg | Get-Content -Stream meta.json

{"_fileName":"61lLJ85GYXL._AC_SL1000_.jpg","last-modified":"2019-10-30T16:28:38+00:00","file-name":"61lLJ85GYXL._AC_SL1000_.jpg","client-name":"j2i.net"}

PS C:\temp\streams>

Next, we want to modify the program to load this metadata if it exists and grab the LastModified property. This is all we need. We are going to use this information to detect if the file has been modified.

void RefreshMetadata()
{
    DirectoryInfo cacheDataDirectory = new DirectoryInfo(Settings.Default.CachePath);
    if (!cacheDataDirectory.Exists)
        return;
    foreach(var file in Files)
    {
        var fileInfo = new FileInfo(Path.Combine(cacheDataDirectory.FullName, file.FileName));
        if (!fileInfo.Exists)
            continue;
        //Great! The file exists! Let's load the metadata for it!
        var metaFilePath = $"{fileInfo.FullName}:meta.json";
        var fileHandle = NativeMethods.CreateFileW(metaFilePath, NativeConstants.GENERIC_READ,
                            0,//NativeConstants.FILE_SHARE_WRITE,
                            IntPtr.Zero,
                            NativeConstants.OPEN_ALWAYS,
                            0,
                            IntPtr.Zero);
        using (StreamReader sr = new StreamReader(new FileStream(fileHandle, FileAccess.Read)))
        {
            var metaString = sr.ReadToEnd();
            var readFileData = JsonConvert.DeserializeObject<FileData>(metaString);
            file.ServerLastModifiedDate = readFileData.ServerLastModifiedDate;
        }

    }
}

The previous code that we wrote needs a few changes. If the file being downloaded has a last modified date, add that to the request in a header field named If-Modified-Since. Thankfully, .Net can convert the DateTimeOffset object to the string format that we need for the request. 

 if(file.ServerLastModifiedDate.HasValue)
 {
     request.Headers.Add("If-Modified-Since", file.ServerLastModifiedDate.Value.ToString("R"));
 }

When the response comes back, we must examine the response code. If the file has been updated the response code will have a response code of 200 (OK). This is the normal response code that we get when we first access a file. If the file has not been updated since the value we pass in If-Modified-Since the response code will be 304 (not modified). The response will have no content. We can move on from this file.

var response = await client.SendAsync(request);
if(response.StatusCode == System.Net.HttpStatusCode.NotModified)
{
    continue;
}

I can’t modify the images on Amazon for testing the behaviour of the app when the image is updated. If you want to test that, you will have to modify the sample program to point to a set of images that you can control to test that out. The NodeJS based http-server utility is useful here if you want to use a random set of images on your local computer for this purpose.

As always, the code for this post is available on GitHub. You can find it in the following repository.

Posts may contain products with affiliate links. When you make purchases using these links, we receive a small commission at no extra cost to you. Thank you for your support.

Twitter: @j2inet
Instagram: @j2inet
Facebook: j2inet
YouTube: j2inet
Telegram: j2inet