Using My Phone as a Web Server – Introduction

I’m back from a recent trip out of the country. While the facility I was staying at would have an Internet connection, the price for Internet access for about a week was a little over 100 USD. I’d rather go without a connection. While I didn’t have access to the Internet, I did have access to a local network. I considered options on how to bring media with me. Rather than bring a few movies and songs here and there I wanted to indiscriminately copy what I could to a drive. In addition to myself, there were three other people with me that might also want to view the media. It made since to host the media on a pocket-sized web server. I setup a Raspberry Pi to do just this and took it with me on the trip.

After the trip was over, I thought to myself that there should be a way to do the same thing in a more compact package. I started to look at what the smallest Raspberry Pi Computer Module based setup would look like, and as I mentally constructed a solution in my mind, I realized it was converging to the same form factor as a phone. I’ve got plenty of old phones lying about. While I wouldn’t suggest this as a general solution (phones are a lot more expensive than a Pi) it is what I decided to have fun with.

Extra, unused Android devices.

There are various ways to run NodeJS on a phone and some other apps in the app store that let you host a web server on your phone. I didn’t use any of these. I am reinventing the wheel simply because I find enjoyment in creating. It was a Sunday night, I was watching my TV lineup, and decided to make a simple proof of concept. I only wanted the PoC to listen for incoming request and send a hard coded HTML page back to the client. I had that working in no time! I’ll build upon this to give it the ability to host static files and media files when I do a future update on this. I’m taking a moment to talk about how I build this first.

I created a new Android project. Before writing code, I declared a few permissions. I like to do this first so that later on I don’t have to wonder why a specific call failed. The permissions I added are for Internet access, accessing the WiFi state, and access to the Wake Lock to keep the device from completely suspending. For what I show here, only Internet capabilities are going to be used. You can choose to omit the other two permissions for this version of the program.

With the permissions in place, I started writing the code. There are only three classes used in the web server (counting an interface as a class).

  • WebServer – Listens for incoming request and passes them off to be handled as they com in
  • ClientSocketHandler – Processes an incoming class and gives the response
  • IStatusUpdater – used for passing status information back to the UI

The WebServer class accepts in its constructor the port on which it should run and a Context object, which is needed for some other calls. A WebServer instance does not begin to listen for connections until the start() method is called. Once it is started, the device retrieves the address of the WiFi adapter and creates a socket that is bound to this address. A status message is also sent to the UI so that it can show the device’s IP address. The class then creates the thread that will listen for incoming connections.

In the function listenerThread(), the class waits for an incoming connection. As soon as it receives one, it creates a new ClientSocketHandler with the socket and lets the ClientSocketHandler process the request and immediately goes back to listening for another connection. It doesn’t wait for the ClientSocketHandler to finish before waiting for the next connection.

package net.j2i.webserver.Service
import android.content.Context
import android.net.wifi.WifiManager
import android.text.format.Formatter
import java.net.InetSocketAddress
import java.net.ServerSocket
import java.net.Socket
import kotlin.concurrent.thread
class WebServer {
    companion object {
    }
    val port:Int;
    lateinit var receiveThread:Thread
    lateinit var  listenerSocket:ServerSocket;
    var keepRunning = true;
    val context:Context;
    var statusReceiver:IStatusUpdateReceiver
    constructor(port:Int, context: Context) {
        this.port = port;
        this.context = context;
        this.statusReceiver = object : IStatusUpdateReceiver {
            override fun updateStatus(ipAddress: String, clientCount: Int) {
                fun updateStatus(ipAddress: String, clientCount: Int) {
                }
            }
        }
    }
    fun start() {
        keepRunning = true;
        val wifiManager:WifiManager =
            this.context.getSystemService(Context.WIFI_SERVICE) as WifiManager;
        val wifiIpAddress:String = Formatter.formatIpAddress(wifiManager.connectionInfo.ipAddress);
        this.statusReceiver.updateStatus(wifiIpAddress, 0)
        this.listenerSocket = ServerSocket();
        this.listenerSocket.reuseAddress = true;
        this.listenerSocket.bind(InetSocketAddress(wifiIpAddress, this.port))
        this.receiveThread = thread(start = true) {
                this.listenerThread()
        }
        //this.receiveThread.start()
    }
    fun listenerThread() {
        while(keepRunning) {
            var clientSocket: Socket = this.listenerSocket.accept()
            val clientSocketHandler = ClientSocketHandler(clientSocket)
            clientSocketHandler.respondAsync()
        }
    }
}

In ClientSocketHandler, the class grabs the input stream (to read the request from the remote client) and the OutputStream (to send data back to the client). Now I haven’t implemented the HTTP protocol. But in HTTP, the client will send a one or more lines that make up the request followed by a blank line. For now, my client handler reads from input stream until that blank line is encountered. Once received, it composes a response.

I’ve got the HTML string that the client is going to return hardcoded into the application. In the response string is converted to a byte array. The size of this array is needed for one of the response headers. The client will receive the size of the response in the header Content-Length. The header for the response is constructed as a string and converted to a byte array. Then the two arrays are sent back to the client (first the header, then the content). After the response is sent, the client has done its work.

package net.j2i.webserver.Service
import android.util.Log
import java.lang.StringBuilder
import java.net.Socket
import kotlin.concurrent.thread
class ClientSocketHandler {
    companion object {
        val TAG = "ClientSocketHandler"
    }
    private val clientSocket: Socket;
    private val responseThread:Thread
    constructor(sourceClientSocket:Socket) {
        this.clientSocket = sourceClientSocket;
        this.responseThread = thread( start = false) {
                this.respond()
        }
    }
    public fun respondAsync() {
        this.responseThread.run()
    }
    private fun respond() {
        val inputStream = this.clientSocket.getInputStream()
        val outputStream = this.clientSocket.getOutputStream()
        var requestReceived = false;
        while(inputStream.available()>0 &&  !requestReceived) {
            val requestLine = inputStream.bufferedReader().readLine()
            Log.i(ClientSocketHandler.TAG, requestLine)
            if(processRequestLine(requestLine)) {
            requestReceived = true;}
        }
        val sb:StringBuilder = StringBuilder()
        val sbHeader = StringBuilder()
        sb.appendLine(
            "<html>"+
                    "<head><title>Test</title></head>" +
                    "<body>Test Response;lkj;ljkojiojioijoij</body>"+
                   "</html>")
        sb.appendLine()
        val responseString = sb.toString()
        val responseBytes = responseString.toByteArray(Charsets.UTF_8)
        val responseSize = responseBytes.size
        sbHeader.appendLine("HTTP/1.1 200 OK");
        sbHeader.appendLine("Content-Type: text/html");
        sbHeader.append("Content-Length: ")
        sbHeader.appendLine(responseSize)
        sbHeader.appendLine()
        val responseHeaderString = sbHeader.toString()
        val responseHeaderBytes = responseHeaderString.toByteArray(Charsets.UTF_8)
        outputStream.write(responseHeaderBytes)
        outputStream.write(responseBytes)
        outputStream.flush()
        outputStream.close()
    }
    fun processRequestLine(requestLine:String): Boolean {
        if(requestLine == "") {
            return true;
        }
        return false;
    }
}

The interface that I mentioned, IStatusUpdateReceiver, is currently only being used to communicate the IP address on which the server is listening back to the UI.

package net.j2i.webserver.Service
interface IStatusUpdateReceiver {
    fun updateStatus(ipAddress:String, clientCount:Int);
}

Since the server runs on a different thread, before updating the UI I must make sure that UI related calls are being performed on the main thread. If you look in the class for MainActivity you will see that I created the  WebServer instance in the activity. I’m only doing this because it is a PoF. If you make your own application, implement this as a service.  I set the statusReceiver member of the WebServer to an annonymous class instance that does nothing more than update the IP address displayed in the UI. The call to set the text in the UI is wrapped in a runOnUiThread block. After this is setup, I call start() on the webserver to get things going.

package net.j2i.webserver
import androidx.appcompat.app.AppCompatActivity
import android.os.Bundle
import android.widget.TextView
import net.j2i.webserver.Service.IStatusUpdateReceiver
import net.j2i.webserver.Service.WebServer
class MainActivity : AppCompatActivity() {
    lateinit var webServer:WebServer
    lateinit var txtIpAddress:TextView
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        this.txtIpAddress = findViewById(R.id.txtIpAddress)
        this.webServer = WebServer(8888, this)
        this.webServer.statusReceiver = object:IStatusUpdateReceiver {
            override fun updateStatus(ipAddress:String, clientCount:Int) {
                runOnUiThread {
                    txtIpAddress.text = ipAddress
                }
            }
        }
        this.webServer.start()
    }
}

I was happy that my proof of concept worked. I haven’t yet decided if I am going to throw this away or continue working from this. In either case, there are a few things that I want to have in whatever my next version is. I do absolutely no exception handling or cleanup in this code. It needs to be able to timeout a connection and refuse connections if it gets inundated. I also want my next version to do actual processing of the incoming HTTP request and serve up content that has been saved to the device’s memory, such as a folder on the device’s memory card. While I am making this to serve up static content, I might add a few services to the server/device side, such as a socket server. That will require a lot more thought.

Twitter: @j2inet
Instagram: @j2inet
Facebook: j2inet
YouTube: j2inet

The James Webb Telescope Has Launched!

On 25 December 2021, the James Webb Telescope (JWT) launched. This last month up to the launch had a couple of delays due to weather and an incident for which they had to ensure there was no damage. At the time that I am writing this, the JWT has not yet been brought up to full operation. But thus far, things have been going well. The JWT is often thought of as the successor to the Hubble telescope. Some call it a replacement, but its capabilities are not identical to that of Hubble. It was designed based on some of the findings of Hubble. I’ve got some readers whose living memory does not go back as far as the Hubble telescope. Let’s take a brief walk-through history.

Rendering of the James Webb Telescope

Edwin Hubble (the person, not the telescope) is most well-known for his astronomical observations and discoveries. Some of his discovers included that there were galaxies beyond the Milky Way, found methods to gauge cosmic distances, and discovered that the further aware from earth that an observed galaxy is, the faster that it is moving away from other galaxies (this is known as “Hubble’s Law”). Edwin Hubble performed many of his observations using what was then the world’s largest telescope, named after James D. Hooker. Naming large telescopes after people was a bit off a tradition.

The Hubble Telescope

Space telescopes were proposed in the early 1920s. As is the case with many high investment scientific endeavors, Hubble’s planning was a joint venture that crossed international borders. The USA’s NASA and the European Space Agency both made contributions to Hubble. The project was started in the 1970s with plans to launch in 1983. There were delays that prevented this. But it finally launched in 1990. Much to the disappointment of many, after launch it was discovered that the Hubble’s main mirror was incorrectly manufactured; the telescope was taking distorted images. It was possible to use software to make some corrections in the image, but servicing was needed to correct the problem. Hubble, being positioned in low earth orbit, was accessible to astronauts by way of the space shuttle. A few years after its launch in 1993 a servicing mission corrected the optical problems. Through several other missions Hubble was maintained and upgraded until 2009. The telescope had been used for over 30 years. The telescope is still partially operational now. Some of the gyroscopes have failed as has one of the high-resolution cameras. But some other cameras and instruments are still operational. A near-Infared telescope is functional but remains offline for the time being. It is expected to be able to maintain functionality until 2040.

The Plane Uranus as seen in Near Infared

While Hubble was operating in its earlier years, plans for its successor had begone. Planning for the James Web Telescope began about 1996. The year prior, in 1995, was the Hubble Deep Field photograph. The Hubble telescope was aimed at a dark patch of sky and took a long exposure photograph. For 10 days the telescope collected whatever bits of light that it could. The result was an image that was full of galaxies! Around 10,000 galaxies were observed through the deep field imaging. Visible, infrared, and ultraviolet wavelengths were used in the imaging.

Hubble Deep Field Image

Earlier I mentioned Edwin Hubble’s discovery of how galaxies further aware are recessing from earth at a faster rate than ones that are closer. The faster the galaxy is moving away, the more red-shifted the light from it is. Red shifting is a form of the doppler effect observed on light. Just as the pitch of a sound will be higher in pitch if it is moving toward and observer and lower in pitch when it is moving away, visible light shifts to become red if the source is moving away from an observer and blue if it is moving closer. Part of the purpose of the JWT is to make observations of astronomical bodies much more distant than the Hubble could. Since these bodies will be more red shifted, the JWT was designed to be sensitive to light that is red shifted. While both the Hubble Telescope and JWT have infrared capabilities, the JWT is designed to see light that is much more red. Because of this goal, the JWT has some rather unusual elements of design and constraints.

Objects radiate their heat out as electromagnetic waves. For objects that are hot enough, we are able to see this radiation as light; a hot piece of metal may glow red or orange. Objects with no glow in visible light may still give off light in the Infared spectrum. Such objects include the earth and the moon, which reflect infrared from the sun and emits heat.

Infrared Photo showing heat leakage from a house

The Hubble was positioned in low earth orbit, about 570km above earth. The moon is about 385,000 km from earth. To avoid the glow of the earth and moon, the JWT is much further aware at 1,500,000 km. The Hubble was in orbit around the earth, but the JWT isn’t really in orbit. It is in a Lagrange point. Objects positioned in a Lagrange point tend to stay in position with very little active adjustments needed.

HST, Webb distance 
  graphic
Relative distances from earth. Image from NASA.gov.

The telescope is still exposed to the sun, which would potentially heat the telescope up and cause the telescope to have its own glow that would interfere with imaging. To prevent the sun from being a problem, the telescope has a multilayered shield on the portion that is facing the sun. The shield is designed to reflect light away and to dissipate heat before it reaches the imaging elements of the telescope. Another unique element of the telescope is the exposed reflector. The reflector is composed of several hexagon-shaped mirrors coated in gold. Gold reflects infrared light very well. Using hexagon segments for the mirror simplifies manufacturing and allows the elements to be more easily folded; the telescope was launched in a fairing with the mirror folded and the sunshield sandwiched over the mirror.

Folded James Webb Telescope.

The JWT’s field of vision is much wider than that of Hubble. It collects about 15 times more light than the Hubble and has a wider field of view. The telescope’s look stands out in that there is no tube wrapped around the optical elements. Optical tubes on terrestrial telescopes protect the elements from debris and stray light. Because of the telescope’s sun shield and its position, it won’t be exposed to stray light from the sun. I’ve not been able to find references on any concern for the mirror being exposed to debris in space (despite being a hard vacuum, it isn’t without debris) but unlike on earth, there are not concerns with it collecting dust. With these differences in design and capabilities and design, what are the plans on how this telescope will be used?

Webb and Hubble mirror 
comparison
Comparison of Hubble and JWT mirror size, from NASA.gov

While I’m not a fan of this description, I often see its purposed summarized as “looking back in time.” Despite my dislike of this description, it isn’t inaccurate. Light takes time to travel. If you look toward the moon, the light reflected from the moon took 3 seconds to travel to your eyes. You are seeing how the moon looked three seconds ago. For the sun, it’s eight minutes ago. These bodies to change dramatically enough for the delay to make a significant difference. But as we look at bodies that are further away, the time it takes to travel becomes more significant. From Mars to earth is about 22 minutes. Jupiter to earth is about 48 minutes. It takes a few hours for light to travel between Pluto and earth. For other galaxies, light takes years. While light-years is a unit of distance, it also tells you how long it takes for light to travel from a body. The JWT’s light collection capabilities make it capable of seeing light far enough aware to collect information on the earlier universe. The Hubble telescope was able to collect information on the universe from about ~13.4 billion years ago while the James Webb Telescope is expected to collect data from about 13.7 billion years ago. That 300,000,000 difference

As of yet, the James Webb Telescope hasn’t taken its first image. This is about 4 days after launch. It has deployed the sun shield. It will take about another 25 days for the telescope to reach its intended position. Before then, the mirror segments must be unfolded into place. If you are waiting to see images from the JWT, it will be a while. There’s calibration and preparation needed. Other than test images, we might not start seeing full images for another six months.

If you want to keep track of where the telescope is and its status, NASA has a site available showing the tracking data.

James Webb Telescope Tracking Site

Developments on the James Webb Telescope will be slow to come at first, but it should be interesting.

Audio version of this podcast.

Alternate File Streams::Security Concerns?

I previously wrote about alternate data streams. Consider this an addendum to that post.

Alternate file streams are a Windows file system feature that allows additional sets of data to be attached to a file. Each data stream can be independently edited. But they are all part of the same file. Since the Windows UI doesn’t show information on these files, it raises a few perceivable security concerns. Whether it is the intention or not, information within an alternate file stream is concealed from all accept those that know to look for it. This isn’t limited to humans looking at the file system, but also other security products that may scan a file system.

It is possible to put executable content within an alternate file stream. Such executable content can’t be invoked directly from the UI, but it can be invoked through tools such as WMI. Given these security concerns that alternate streams may raise, why do I still use them? Those concerns are only applicable to how other untrusted entities may use the feature. But any action of an untrusted entity may be one of concerns.

I thought this concern was worth mentioning because if you try searching for more information on alternate file streams, these concerns are likely to come up on the first page of results.

Working With Alternative Data Streams::The “Hidden” Part of Your Windows File System on Windows

In the interest of keeping a cleaner file system, I sometimes try to minimize the number of files that I need to keep data organized. A common scenario where this goal is expressed is when writing an application that must sync with some other data source such as a content management system. Having a copy of the files from another system isn’t always sufficient. Sometimes additional data is needed for keeping the computers in a solution in sync. For a given file, I may need to also track an etag, CRC, or information on the purpose of a file. There are some common solutions for organizing this data. One is to have one additional file that contains all of the additional meta data for the files being synced. Another is to make an additional data file for each content file that contains this information. The solution that I prefer isn’t quite either of these. I prefer to have the data within an “alternate stream” of the same file. If the file get’s moved elsewhere on the filesystem, the additional data will move with it.

This is very much a Windows-Only solution. This will only work on the NTFS file system. If you attempt to access an alternative stream on a FAT32 file system, it will fail since that file system does not support them.

I most recently used this system of organization when I inherited a project that was, in my opinion, built with the wrong technology. In all fairness, many features of the application in question were implemented through scope creep. I reimplemented the application in about a couple of days using .Net technologies (this was much easier for me to do since, unlike the original developers, I had the benefit of having a complete requirements). There were a lot of aspects of that project that will be expressed in posts in the coming weeks.

The reason that I call this feature “hidden” is because the Windows UI does not give any visual indicator that a file has an additional data stream. There is no special icon. If you check on the size of a file, it will only show you the size of the main data stream. (In theory, one could add 3 gigs of data to the secondary stream of a 5 byte file and the OS would only report the file as being 5 bytes in size).

I’ll demonstrate accessing this stream from within the .Net Framework. There’s not built-in support for the alternative datastreams, but there is built in support for Windows file handles. Using a P/Invoke you can get a Windows file handle and then pass it to a .Net FileStream object to be used with all the other .Net features.

Every file has a default data stream. This is the stream you would see as being the normal file. It contains the file data that you are usually working with. With our normal concept of files and directories, files contain data and directories contain files, but no data directly. A file can contain any number of alternative data streams. Each one of these streams has a name of your choosing. Directories can have alternative data streams too!

To start experimenting with streams, you only need the command prompt. Open the command prompt and navigate to a directory in which you will place your experimental streams. type the following.

echo This is data for my alternative stream > readme.txt:stream

If you get a directory listing, we find that the files is listed as zero bytes in size.

c:\temp\streams>echo This is data for my alternative stream > readme.txt:stream

c:\temp\streams>dir
 Volume in drive C has no label.
 Volume Serial Number is 46FF-0556

 Directory of c:\temp\streams

10/11/2021  11:22 AM    <DIR>          .
10/11/2021  11:22 AM    <DIR>          ..
10/11/2021  11:22 AM                 0 readme.txt
               1 File(s)              0 bytes
               2 Dir(s)  108,162,564,096 bytes free

c:\temp\streams>

Is the data really there? From the command line, we can view the data using the more command (the type command doesn’t accept the syntax needed to refer to the stream).

c:\temp\streams>more < readme.txt:stream
This is data for my alternative stream

Windows uses alternative data streams for various system purposes. There are a number of names that you may encounter in files that Windows manages. This is a list of some well known stream names.

  • $DATA – This is the default stream. This stream contains the main (regular) data for a file. If you open README.TXT, this has the same effect as opening README.TXT:$DATA.
  • $BITMAP – data used for managing a b-tree for a directory. This is present on every directory.
  • $ATTRIBUTE_LIST – A list of attributes for a file.
  • $FILE_NAME – Name of file in unicode characters, including short name and hard links
  • $INDEX_ALLOCATION – used for managing large directories

There are some other names. In general, with the exception of $DATA, I would suggest not altering these streams.

Windows does give you the ability list alternative streams through PowerShell. We will look at that in a moment. For now, let’s say you had to make your own tool for managing such resources. The utility of this example is it is giving us an opportunity to see how we might work with these resources in code. One of the first tools that I think is useful would be a command line tool that would transfer data from one stream to another. With this tool, I can read from a stream and either write it to a console or to another stream. The only thing that affects where it is written is the file name. It only took a few minutes to write such a tool using C++. It is small enough to put the entirety of the code here.

#include <Windows.h>
#include <iostream>
#include <string>
#include <vector>
#include <list>

using namespace std;

const int BUFFER_SIZE = 1048576;

int main(int argc, CHAR ** argv)
{
    wstring InputStreamName = L"";
    wstring OutputStreamName = L"con:";
    wstring InputPrefix = L"--i=";
    wstring OutputPrefix = L"--o=";

    wstring Instructions =
        L"To use this tool, provide an input and an output file for it. The syntax looks like the following.\r\n\r\n"
        L"StreamStreamer --i=inputFileName --o=OutputFileName.ext::streamName\r\n\r\n";

    HANDLE hInputFile = INVALID_HANDLE_VALUE;
    HANDLE hOutputFile = INVALID_HANDLE_VALUE;

    vector<wstring> arguments(argc);

    for (auto i = 0; i < argc; ++i)
    {
        auto arg = string(argv[i]);
        arguments[i] = (wstring(arg.begin(), arg.end()));
    }

    for (int i = 0; i < argc; ++i)
    {
        if (!arguments[i].compare(0, InputPrefix.size(), InputPrefix))
            InputStreamName = arguments[i].substr(InputPrefix.size());
        if (!arguments[i].compare(0, OutputPrefix.size(), OutputPrefix))
            OutputStreamName = arguments[i].substr(OutputPrefix.size());
    }

    if ((!InputStreamName.size()) || (!OutputStreamName.size()))
    {
        wcout << Instructions;
        return 0;
    }

    hInputFile = CreateFile(InputStreamName.c_str(), GENERIC_READ, FILE_SHARE_READ, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
    if (hInputFile != INVALID_HANDLE_VALUE)
    {
        hOutputFile = CreateFile(OutputStreamName.c_str(), GENERIC_WRITE, FILE_SHARE_READ, 0, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, 0);
        if (hOutputFile != INVALID_HANDLE_VALUE)
        {
            vector<char> buffer = vector<char>(BUFFER_SIZE);
            DWORD bytes_read = 0;
            DWORD bytes_written = 0;
            do {
                bytes_read = 0;
                if (ReadFile(hInputFile, &buffer[0], BUFFER_SIZE, &bytes_read, 0))
                    WriteFile(hOutputFile, &buffer[0], bytes_read, &bytes_written, 0);
            } while (bytes_read > 0);
            CloseHandle(hOutputFile);
        }
        CloseHandle(hInputFile);
    }
}


Usage is simple. The tool takes two arguments; an input stream name and an output stream name are passed prefixed with either --i= or --o=. If no output name is specified, it defaults to an output name of con:. This name, con:, refers to a console. That had been a reserved file name for the console. I have the vague idea there may be some other console name, but could not find it. con: is a carry-over of the DOS days of 30+ years ago. It worked then, and it still works now. So I’m sticking with it. Note that there is

After compiling this, I can use it to retrieve the text that I attached to the stream earlier.

c:\temp\streams>StreamStreamer.exe --i=readme.txt:Stream
This is data for my alternative stream

c:\temp\streams>

I can also use it to take the contents of some other arbitrary file and attach it to an existing file in an alternative stream. In testing, I took a JPG I had of the moon and attached it to a file. Then I extracted it from that alternative stream and wrote it to a different regular file just to ensure that I had an unaltered data stream.

c:\temp\streams>StreamStreamer.exe --i=Moon.JPG --o=readme.txt:moon

c:\temp\streams>StreamStreamer.exe --i=readme.txt:moon --o=m.jpg

c:\temp\streams>dir *.jpg
 Volume in drive C has no label.
 Volume Serial Number is 46FF-0556

 Directory of c:\temp\streams


10/12/2021  02:54 PM         3,907,101 m.jpg
12/21/2020  07:46 PM         3,907,101 Moon.JPG
               3 File(s)      7,814,202 bytes
               0 Dir(s)  105,063,383,040 bytes free

c:\temp\streams>

You will probably want the ability to see what streams are inside of a file. You could download the Streams tool from System Internals, or you could use PowerShell. PowerShell has built in support for streams. I’ll be using that throughout out the rest of this writeup. To view streams with PowerShell, use the Get-Item command with the -stream * parameter.

PS C:\temp\streams> Get-Item .\readme.txt -stream *


PSPath        : Microsoft.PowerShell.Core\FileSystem::C:\temp\streams\readme.txt::$DATA
PSParentPath  : Microsoft.PowerShell.Core\FileSystem::C:\temp\streams
PSChildName   : readme.txt::$DATA
PSDrive       : C
PSProvider    : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName      : C:\temp\streams\readme.txt
Stream        : :$DATA
Length        : 15

PSPath        : Microsoft.PowerShell.Core\FileSystem::C:\temp\streams\readme.txt:moon.jpg
PSParentPath  : Microsoft.PowerShell.Core\FileSystem::C:\temp\streams
PSChildName   : readme.txt:moon.jpg
PSDrive       : C
PSProvider    : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName      : C:\temp\streams\readme.txt
Stream        : moon.jpg
Length        : 3907101

PSPath        : Microsoft.PowerShell.Core\FileSystem::C:\temp\streams\readme.txt:stream
PSParentPath  : Microsoft.PowerShell.Core\FileSystem::C:\temp\streams
PSChildName   : readme.txt:stream
PSDrive       : C
PSProvider    : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName      : C:\temp\s\readme.txt
Stream        : stream
Length        : 43



PS C:\temp\streams>

If you are making an application that uses alternative streams, you will want to know how to list the streams from within it also. That is also easy to do. Since the much beloved Windows Vista we’ve had a Win32 API for enumerating streams. The functions FindFirstStreamW/FindFirstStreamTransactedW and FindNextStreamW will do this for you. Take note that there only exist Unicode versions of these functions. ASCII variations are non-existent. If you have ever used FindFirstFile or FindNextStreamW the usage is similar.

Two variables are needed to search for streams. One variable is a HANDLE that is used as an identifier for the resources and state of the search request. The other is a WIN32_FIND_STREAM_DATA structure into which data on streams that were found are put. FindFirstStreamW will return a handle and populate a WIN32_FIND_STREAM_DATA with the first stream it finds. From there, each time FindNextStreamW is called with the HANDLE that had been returned earlier, it will populate a WIN32_FIND_STREAM_DATA with the information on the next stream. When no more streams are found, FindNextStreamW will have a return value of ERROR_HANDLE_EOF.

#include <Windows.h>
#include <iostream>
#include <vector>
#include <string>

using namespace std;

int main(int argc, char**argv)
{
    WIN32_FIND_STREAM_DATA fsd;
    HANDLE hFind = NULL;
    vector<wstring> arguments(argc);

    for (auto i = 0; i < argc; ++i)
    {
        auto arg = string(argv[i]);
        arguments[i] = (wstring(arg.begin(), arg.end()));
    }

    if (arguments.size() < 2)
        return 0;
    wstring fileName = arguments[1];

    try {
        hFind = FindFirstStreamW(fileName.c_str(), FindStreamInfoStandard, &fsd, 0);
        if (hFind == INVALID_HANDLE_VALUE) throw ::GetLastError();
        const int BUFFER_SIZE = 8192;
        WCHAR buffer[BUFFER_SIZE] = { 0 };
        WCHAR fileNameBuffer[BUFFER_SIZE] = { 0 };

        wcout << L"The following streams were found in the file " << fileName << endl;
        for (;;)
        {
            swprintf(fileNameBuffer, BUFFER_SIZE, L"%s%s", fileName.c_str(), fsd.cStreamName);
            swprintf_s(buffer,BUFFER_SIZE, L"%-50s %d", fileNameBuffer, fsd.StreamSize);
            wstring formattedDescription = wstring(buffer);
            wcout << formattedDescription << endl;

            if (!::FindNextStreamW(hFind, &fsd))
            {
                DWORD dr = ::GetLastError();
                if (dr != ERROR_HANDLE_EOF) throw dr;
                break;
            }
        }
    }
    catch (DWORD err)
    {
        wcout << "Oops, Error happened. Windows error number " << err;
    }
    if (hFind != NULL)
        FindClose(hFind);
}

For my actual application purposes, I don’t need to query the streams in a file. The streams of interest to me will have a predetermined name. Instead of querying for them, I attempt to open the stream. If it isn’t there, I will get a return error code indicating that the file isn’t there. Otherwise I will have a file HANDLE for reading and writing. With what I’ve written so far, you could begin using this feature in C/C++ immediately. But my target is the .Net Framework. How do we use this information there?

In Win32, you can read or write these alternative data streams as you would any other file by using the correct stream name. If you try that within the .Net Framework, it won’t work. Before even hitting the Win32 APIs, the .Net Framework will treat the stream name as an invalid file name. To work around this, you’ll need to P/Invoke the Win32 API for opening files. Thankfully, once you have a file handle, the .Net Framework will work with that file handle just fine and allow you to use all the methods that you would with any other stream.

Before adding the P/Invoke that are needed to use this functionality in .Net, let’s defined a few numerical constants.

    public partial class NativeConstants
    {
        public const uint GENERIC_WRITE = 1073741824;
        public const uint GENERIC_READ = 0x80000000;
        public const int FILE_SHARE_DELETE = 4;
        public const int FILE_SHARE_WRITE = 2;
        public const int FILE_SHARE_READ = 1;
        public const int OPEN_ALWAYS = 4;
    }

These may look familiar. These constants have the same names as constants that were used in C when calling the Win32 API. These constants, as their names suggest, are used to indicate the mode in which files should be opened. Now fot the P/Invokes to the calls to open the files.

    public partial class NativeMethods
    {
        [DllImportAttribute("kernel32.dll", EntryPoint = "CreateFileW")]
        public static extern System.IntPtr CreateFileW(
            [InAttribute()][MarshalAsAttribute(UnmanagedType.LPWStr)] string lpFileName,
            uint dwDesiredAccess,
            uint dwShareMode,
            [InAttribute()] System.IntPtr lpSecurityAttributes,
            uint dwCreationDisposition,
            uint dwFlagsAndAttributes,
            [InAttribute()] System.IntPtr hTemplateFile
        );

    }

That’s it! That is the only P/Invoke that is needed.

The data that I was writing to these files was metadata on on files for matching them up with entries in a CMS. This includes information like the last date that it was updated on the CMS, a CRC or ETAG for knowing if the version on the local computer is the same as the one on the CMS, and a title for presenting to the user (Which may be different than the file name itself). I’ve decided that the name of the stream in which I am placing this data to simply be meta. I’m using JSON for the data encoding. For your purposes, you could use any format that fits your application. Let’s open a stream for writing.

I’ll use the Win32 CreateFileW function to get a file handle. That handle is passed to the .Net FileStream constructor. From there, there is no difference in how I would read or write from this

var filePath = Path.Combine(ds.DownloadCachePath, $"{fe.ID}{extension}");
FileInfo fi = new FileInfo(filePath);
var fullPath = fi.FullName;
if (fi.Exists)
{
    var metaStream = NativeMethods.CreateFileW(
        $"{fullPath}:meta",
        NativeConstants.GENERIC_READ,
        NativeConstants.FILE_SHARE_READ,
        IntPtr.Zero,
        NativeConstants.OPEN_ALWAYS,
        0,
        IntPtr.Zero);
    using (StreamReader sr = new StreamReader(new FileStream(metaStream, FileAccess.Read)))
    {
        try
        {
            var metaData = sr.ReadToEnd();
            if (!String.IsNullOrEmpty(metaData))
            {
                var data = JsonConvert.DeserializeObject<FileEntry>(metaData);
                fe.LastModified = data.LastModified;
            }
        } catch(IOException exc)
        {
        }
    }
}

I said earlier that this is a Windows-Only solution and that it doesn’t work on the Fat32 file system. The two implications of this is that if you are using this in in a .Net environment that is running on another operating system this won’t work. It will likely fail since the P/Invokes won’t be able to bind. The other potential problem demands an active check within the code. If a a program using alternative file streams is given a FAT32 file system to work with, it should detect that it is on the wrong type of file system before trying to perform actions that will fail. Detecting the file system type only requires a few lines of code. In .Net, the following code will take the path of the currently running assembly, see what drive it is on, and retrieve the file system type.

 String assemblyPath = typeof(FileSystemDetector).Assembly.Location;
 String driveLetter = assemblyPath[0].ToString();
 DriveInfo driveInfo = new DriveInfo(driveLetter);
 string fsType = driveInfo.DriveFormat;
 return fsType;

If this code is run on a a drive using the NTFS file system, the return type will be the string value NTFS. If it is anything else, know that attempts to access alternative streams will fail. If you try to copy these file to a FAT32 drive, Windows will warn you of a loss of data. Only the default streams will be copied to the FAT32 drive.

In the next posts on this I will demonstrate a practical use. I’ll also talk about what some might see as a security concern with alternate file streams.

Using an iPhone as a Beacon

As with any project, I had a list of milestones and dates on which I expected to hit them leading up to project completion. One of the elements for the project was an application that needed to detect its proximity to other devices to select a device for interaction. I planned to use iBeacons for this, and had a delivery date on some beacons for development. The delivery date came, a box with the matching tracking number came, but there were no iBeacons inside. Instead, there was a phone case. This isn’t the first time I have ordered one item and Amazon has sent me another. I went online and filled out a form to have the order corrected. They stated I would have the item in another 5 days. In the mean time, I didn’t want to let progress slip. I’ve heard several times “You can use an iPhone as an iBeacon.” I now had motivation to look into this. You can in fact use a phone as an iBeacon. But you have to write an application yourself to use it this way.

When I took a quick look in the App store, I couldn’t find an app for this purpose. So I decided to make an application myself. It isn’t hard. In my case, I’m emulating an iBeacon as a stand-in for actual hardware. But there are other reasons you might want to do this. For example, if I were using an iPad as a display showing more information on an exhibit users browsing the exhibit could interact with the content on the display using their own phone. The iBeacon signal could be used so that the user’s phone knows which display it is close to, allowing them to trigger interactions from their own phone (a valuable method of interaction given the higher concerns on hygiene and concerns over shared touch surfaces).

Beacons are uniquely identified by three pieces of data; the UUID, Major number, and Minor number. A UUID, or Universally Unique ID, is usually shared among a group of iBeacons that are associated with the same entity. The usage of the Major and Minor numbers is up to the entity. Usually the Major will be used to group related iBeacons together with the Minor number being used as a unique ID within the the set. I’ll talk more about these numbers in another post.

For my iPhone application, I have created a few variables to hold the simulated Beacon’s identifiers. I also have a variable to track whether the iBeacon is active, and have defined a Zero UUID to represent a UUID that has not been assigned a value.

class BeaconManager {
    
    var objectWillChange = PassthroughSubject<Void, Never>()
    
    let ZeroUUID = UUID.init(uuidString: "00000000-0000-0000-0000-000000000000")
    
    var BeaconUUID = UUID(uuidString: "00000000-0000-0000-0000-000000000000") {
        didSet { updateUI() }
    }
    
    var Major:UInt16 = 100 {
        didSet { updateUI() }
    }
    
    var Minor:UInt16 = 2 {
        didSet { updateUI() }
    }
    
    var IsActive:Bool = false {
        didSet { updateUI() }
    }
}

I am going to use Swift UI for displaying information. That is why setting these variables also triggers a call to updateUI(). There are some callbacks that are made by Apple’s iBeacon API. For these, I’ll need to also need to implement the CBPeripheralManagerDelegate. This protocol is defined in CoreBluetooth. We also need permission for the device to advertise its presence over Bluetooth. permission. Bluetooth is often used for indoor location (which will be my ultimate intention). Let’s get all these other things in place. The necessary import statements and inheritance will look like the following.

import Foundation
import CoreLocation
import CoreBluetooth
import Combine

class BeaconManager: NSObject, CBPeripheralManagerDelegate, Identifiable, ObservableObject {
   ...
}

For the Bluetooth permission that the application needs, a new String value must be added to the Info.plist. The item’s key is NSBluetoothAlwaysUsageDescription. The value should be a text description that will be presented to the user letting them know why the application is requesting Bluetooth permissions.

I want the simulated iBeacon to have the same value every time the application runs. At runtime, the application is going to check whether there is a UUID already saved in the settings. If there is not one, then it will generate a new UUID and save it to the settings. From then on, it will always use the same ID. I do the same thing with the Major and Minor numbers using the UInt16.random(in:) function. This information together is used for create a CLBeaconRegion.

    func createBeaconRegion() -> CLBeaconRegion {
        let settings = UserDefaults.standard
        if let savedUUID = settings.string(forKey: BEACON_UUID_KEY) {
            if let tempBeaconUUID = UUID(uuidString: savedUUID) {
                BeaconUUID = tempBeaconUUID
            }
        }
        if(BeaconUUID == nil){
            BeaconUUID = UUID()
            settings.setValue(BeaconUUID!.uuidString, forKey: BEACON_UUID_KEY)
            settings.synchronize()
        }   
        let majorValue = settings.integer(forKey: BEACON_MAJOR_KEY) ?? 0
        if(majorValue == 0) {
            Major = UInt16.random(in: 1...65535)
            settings.setValue(Major, forKey: BEACON_MAJOR_KEY)
        }   
        let minorValue = settings.integer(forKey: BEACON_MINOR_KEY) ?? 0
        if(minorValue == 0) {
            Minor = UInt16.random(in: 1...65535)
            settings.setValue(Minor, forKey: BEACON_MINOR_KEY)
        }   
        print(BeaconUUID?.uuidString)
        let major:CLBeaconMajorValue = Major
        let minor:CLBeaconMinorValue = Minor
        let beaconID =  "net.domain.application"
        return CLBeaconRegion(proximityUUID:BeaconUUID!, major: major, minor: minor, identifier: beaconID)
    }

When I first tried to use the CLBeaconRegion it failed, and I was confused. After a bit more reading, I found out why. The Bluetooth radio can take a moment to initialize into the mode that the code needs it for. Trying to use it too soon can result in failure. To fix this, wait for a callback to CBPeripheralManagerDelegate::peripheralManagerDidUpdateState(_ peripheral:CBPeripheralManager).In the handler for this callback, check if the .state of the peripheral variable .poweredOn. If it is, then we can start using our CLBeaconRegion. We can call startAdvertising on the CBPeripheralManager object to make the iBeacon visible. When we want the phone to no longer act as an iBeacon, we can call the stopAdvertising. Note that the device will only continue to transmit while the application has focus. If the application gets pushed to the background, the phone sill stop presenting as an iBeacon.

    func peripheralManagerDidUpdateState(_ peripheral: CBPeripheralManager) {
        if(peripheral.state == .poweredOn) {
            let  beaconRegion = createBeaconRegion()
            let peripheralData = beaconRegion.peripheralData(withMeasuredPower: nil)
            peripheral.startAdvertising(((peripheralData as NSDictionary)as! [String:Any]))
            IsActive = true
        }
    }

    func start() {
        if(!IsActive) {
            peripheral = CBPeripheralManager(delegate:self, queue:nil)
        }
    }
    
    func stop() {
        if(IsActive) {
            if (peripheral != nil){
                peripheral!.stopAdvertising()
            }
            IsActive = false
        }
    }

The code for the class I used for simulating the iBeacon follows. For the simplest use case, just instantiate the class and call the start() method. Provided the Info.plist has been populated with a value for NSBluetoothAlwaysUsageDescription and the user has granted permission, it should just work. In the next post, lets look at how to detect iBeacons with an iOS application. The next application isn’t limited to only detecting iPhones acting as iBeacons. It will work with real iBeacons too. As of now I have gotten my hands on a physical iBeacon compatible transmitter. While any iBeacon transmitter should work, if you would like to follow along with the same iBeacon that I am using, you can purchase the following from Amazon (affiliate link).

Posts may contain products with affiliate links. When you make purchases using these links, we receive a small commission at no extra cost to you. Thank you for your support.


Twitter: @j2inet
Instagram: @j2inet
Facebook: j2inet
YouTube: j2inet

High Resolution Video Capture on the Raspberry Pi

I’ve previously talked about video capture on the Raspberry Pi using an HDMI device that interfaced with the camera connector. Today I’m looking at using USB capture devices. USB capture devices often present as web cams. The usual software and techniques that you may use with a webcam should generally work here.

The two devices that I have are the Elgato CamLink 4K and the Atmos Connect. Despite the name, the CamLink 4K does not presently work in 4K mode for me on the Pi. (I’m not sure that the pi would be able to handle that even if it did). I am using a Raspberry Pi 4. I got better results with the Atmos Connect; it is able to send the Pi pre-compressed video so that the pi didn’t have to compress it.

Hardware setup is simple. I connected the USB capture device to the pi and connected an HDMI source to the capture device. If you want to be able to monitor the video while it is captured you will also need an HDMI splitter; the Pi does not show video while it is being captured. Most of what needs to be done will be in the Raspberry Pi terminal.

If you want to ensure that your capture device was detected you can use the lsusb command. This command lists all the hardware detected on the USB port. If you can’t recognize a device there, using it, disconnecting a device, using it again, and noting the difference will let you know identify a line in the output to an item of hardware. Trying first with the Elgato CamLink, my device was easily identified. There was an item labeled as Elgato Systems GmbH.

I’ve not been able to make the devices work with raspistill or raspivid, but it works with ffmpeg and video for linux (v4l-utils). To install Video 4 Linux, use the following command.

sudo apt install v4l-utils

Once Video 4 Linux is installed, you can list the devices that it detects and the device file names.

v4l2-ctl --list-devices

In addition to hardware encoders, I get the following in my output.

Cam Link 4K: Cam Link 4K (usb-0000:01:00.0-2):
          /dev/video0
          /dev/video1

The device name of interest is the first, /dev/video0. Using that file name, we can check what resolutions that it supports with ffmpeg.

$ ffmpeg -f v4l2 -list_formats all -i /dev/video0

[video4linux2,v4l2 @ 0xcca1c0] Raw     :    yuyv422:      YUYV 4:2:2 : 1920x1080
[video4linux2,v4l2 @ 0xcca1c0] Raw     :       nv12:      YUYV 4:2:2 : 1920x1080
[video4linux2,v4l2 @ 0xcca1c0] Raw     :   yuyv420p:      YUYV 4:2:0 : 1920x1080

For this device, the only resolution supported in 1920×1080. I have three options for my format, yuyv422m nv12, and yuyv420p. Before we start recording, let’s view the input. With the following command, we can have ffmpeg read from our video capture device and display the HDMI video stream on the screen.

ffplay -f v4l2 -input_format nv12 -video_size 1920x1080 -framerate 30 -i /dev/video0

In your case, the format (nv12), resolution, and hardware file name may need to be different. If all goes well after running that command, you will see the video stream. Let’s have the video dump to a file now.

ffmpeg -f v4l2 -thread_queue_size 1024 -input_format nv12 -video_size 1920x1080 -framerate 30 -i /dev/video0 -f pulse -thread_queue_size 1024 -i default -codec copy output.avi

This command will send the recorded data to an AVI file. AVI is one of the envelope formats in which audio, video, and other data can be packaged together. You will probably want to convert this to a more portable format. We can also use ffmpeg to convert the output file from AVI to MP4. I’m going to use H264 for video encoding and AAC for audio encoding.

ffmpeg -i output.avi  -vcodec libx264 -acodec aac -b:v 2000k -pix_fmt yuv420p output.mp4

You can find an audio entry on this blog post on Spotify!

Those of you that follow me on Instagram, you may have seen a picture of the equipment that I use to record the video walkthrough of how to do this. A list of those items used is below. Note that these are Amazon Affiliate links.

BlackMagic Designs ATEM
Mini
Mini Pro
Mini Pro Iso

Visual Studio 2022 Release Date

Visual Studio 2022 has been available in test form for a while. I’ve got the release candidate running now. But there is now a release date available for it. If you are looking to upgrade to VS 2022, your wait will be over on November 8, 2021. On this day Microsoft is holding a launch event for Visual Studio 2022. This isn’t just a software release, but also will have demonstrations on what Visual Studio 2022 is bringing.

Visual Studio 2022 brings greater support/integration with GitHub (Microsoft agreed to purchase GitHub back in 2018), code editor, and debugging improvements. The range of new feature touch the areas of WPF, WinForms, WinUI, ASP.NET, and areas not traditionally thought of as Windows specific, such as cross-platform game technologies, developing applications for Mac, and apps for Linux.

The fun starts on November 8, 8:30AM PDT. Learn more here.

#VS2022


Twitter: @j2inet
Instagram: @j2inet
Facebook: j2inet
YouTube: j2inet

Posts may contain products with affiliate links. When you make purchases using these links, we receive a small commission at no extra cost to you. Thank you for your support.


Simple HTTP Server in .Net

.Net and .Net Core both already provide fully functional HTTPS servers, as does IIS on Windows. But I found the need to make my own HTTP server in .Net for a plugin that I was making for a game (Kerbal Space Program, specifically). For my scenario, I was trying to limit the assemblies that I needed to add to the game to as few as possible. So I decided to build my own instead of adding references to the assemblies that had the functionality that I wanted.

For the class that will be my HTTP server, there are only two member variables needed. One to hold a reference to a thread that will accept my request and another for the TcpListener on which incoming requests will come.

Thread _serverThread = null;
TcpListener _listener;

I need to be able to start and top the server at will. For now, when the server stops all that I want it to do is terminate the thread and release any network resources that it had. In the Start function, I want to create the listener and start the thread for receiving the requests. I could have the server only listen on the loopback adapter (localhost) by using the IP address 127.0.0.1 (IPV4) or :1 (IPV6). This would generally be preferred unless there is a reason for external machines to access the service. I’ll need for this to be accessible from another device. Here I will use the IP address 0.0.0.0 (IPV4) or :0 (IPV6) . I’ll be using

public void Start(int port = 8888)
{
    if (_serverThread == null)
    {
        IPAddress ipAddress = new IPAddress(0);
        _listener = new TcpListener(ipAddress, 8888);
        _serverThread = new Thread(ServerHandler);
        _serverThread.Start();
    }
}

public void Stop()
{
    if(_serverThread != null)
    {
        _serverThread.Abort();
        _serverThread = null;
    } 
}

The TcpListener has been created, but it isn’t doing anything yet. The call to have it listen for a request is a blocking request. The TcpListener will start listening on a different thread. When a request comes in, we read the request that was sent and then send a response. I’ll read the entire response and store it in a string. But I’m not doing anything with the request just yet. For the sake of getting to something that functions quickly, I’m going to hardcode a response

String ReadRequest(NetworkStream stream)
{
    MemoryStream contents = new MemoryStream();
    var buffer = new byte[2048];
    do
    {
        var size = stream.Read(buffer, 0, buffer.Length);
        if(size == 0)
        {
            return null;
        }
        contents.Write(buffer, 0, size);
    } while (stream.DataAvailable);
    var retVal = Encoding.UTF8.GetString(contents.ToArray());
    return retVal;
}

void ServerHandler(Object o)
{
    _listener.Start();
    while(true)
    {
        TcpClient client = _listener.AcceptTcpClient();
        NetworkStream stream = client.GetStream();

        try
        {
            var request = ReadRequest(stream);

            var responseBuilder = new StringBuilder();
            responseBuilder.AppendLine("HTTP/1.1 200 OK");
            responseBuilder.AppendLine("Content-Type: text/html");
            responseBuilder.AppendLine();
            responseBuilder.AppendLine("<html><head><title>Test</title></head><body>It worked!</body></html>");
            responseBuilder.AppendLine("");
            var responseString = responseBuilder.ToString();
            var responseBytes = Encoding.UTF8.GetBytes(responseString);

            stream.Write(responseBytes, 0, responseBytes.Length);

        }
        finally
        {
            stream.Close();
            client.Close();
        }
    }
}

To test the server, I made a .Net console program that instantiates the server.

namespace TestConsole
{
    class Program
    {
        static void Main(string[] args)
        {

            var x = new HTTPKServer();
            x.Start();
            Console.ReadLine();
            x.Stop();
        }
    }
}

I rant the program and opened a browser to http://localhost:8888. The web page shows the response “It worked!”. Now to make it a bit more flexible. The logic for what to do with a response will be handled elsewhere. I don’t want it to be part of the logic for the server itself. I’m adding a delegate to my server. The delegate function will receive the request string and must return the response bytes that should be sent. I’ll also need to know the Mime type. I’ve made a class for holding that information.

public class Response
{
    public byte[] Data { get; set; }
    public String MimeType { get; set; } = "text/plain";
}

public delegate Response ProcessRequestDelegate(String request);
public ProcessRequestDelegate ProcessRequest;

I’m leaving the hard coded response in place, though I am changing the message to say that no request processor has been added. Generally, the code is expecting that the caller has registered a delegate to perform request processing. If it has not, then this will server as a message to the developer.

The updated method looks like the following.

void ServerHandler(Object o)
{
    _listener.Start();
    while(true)
    {
        TcpClient client = _listener.AcceptTcpClient();
        NetworkStream stream = client.GetStream();

        try
        {
            var request = ReadRequest(stream);

            if (ProcessRequest != null)
            {
                var response = ProcessRequest(request);
                var responseBuilder = new StringBuilder();
                responseBuilder.AppendLine("HTTP/1.1 200 OK");      
                responseBuilder.AppendLine("Content-Type: application/json");
                responseBuilder.AppendLine($"Content-Length: {response.Data.Length}");
                responseBuilder.AppendLine();

                var headerBytes = Encoding.UTF8.GetBytes(responseBuilder.ToString());

                stream.Write(headerBytes, 0, headerBytes.Length);
                stream.Write(response.Data, 0, response.Data.Length);
            }
            else
            {
                var responseBuilder = new StringBuilder();
                responseBuilder.AppendLine("HTTP/1.1 200 OK");
                responseBuilder.AppendLine("Content-Type: text/html");
                responseBuilder.AppendLine();
                responseBuilder.AppendLine("<html><head><title>Test</title></head><body>No Request Processor added</body></html>");
                responseBuilder.AppendLine("");
                var responseString = responseBuilder.ToString();
                var responseBytes = Encoding.UTF8.GetBytes(responseString);

                stream.Write(responseBytes, 0, responseBytes.Length);
            }
        }
        finally
        {
            stream.Close();
            client.Close();
        }
    }
}

The test program now registers a delegate. The delegate will show the request and send a response that is derived by the time. I’m marking the response as a JSON response.

static Response ProcessMessage(String request)
{
    Console.Out.WriteLine($"Request:{request}");
    var response = new HTTPKServer.Response();
    response.MimeType = "application/json";
    var responseText = "{\"now\":" + (DateTime.Now).Ticks + "}";
    var responseData = Encoding.UTF8.GetBytes(responseText);
    response.Data = responseData;
    return response;

}

static void Main(string[] args)
{

    var x = new HTTPServer();
    x.ProcessRequest = ProcessMessage;
    x.Start();
    Console.ReadLine();
    x.Stop();
}
}

I grabbed my iPhone and made a request to the server. From typing the URL in there are actually two requests; there is a request for the URL that I typed for for an icon for the site.

Request:
Request:GET /HeyYall HTTP/1.1
Host: 192.168.50.79:8888
Upgrade-Insecure-Requests: 1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/91.0.4472.80 Mobile/15E148 Safari/604.1
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: keep-alive


Request:GET /favicon.ico HTTP/1.1
Host: 192.168.50.79:8888
Connection: keep-alive
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/91.0.4472.80 Mobile/15E148 Safari/604.1
Accept-Encoding: gzip, deflate
Accept-Language: en,en-US;q=0.9,ja-JP;q=0.8

To make sure this work, I loaded it into my game and made a request. The request was successful and I am ready to move on to implementing the logic that is needed for the game.

Why Your Computer Might not Run Windows 11

As of October 5, 2021, the RTM version of Windows 11 is available for download. I’ve downloaded it and have tried to install it on a range of machines. In doing this, the first thing that stands out is that there are a lot of otherwise capable machines in existence today that may not be able to run the new operating system. There are three requirements that tended to be the obstacles on the computers on which I did not install Windows 11.

  • Secure Boot Enabled
  • TPM 2.0
  • Unsupported Processor

Because of the Secure Boot and the TPM requirements, I found that I could not install Windows 11 on my Macs using Bootcamp. Other guides that I’ve found all have Windows 11 on Mac being installed within a virtual machine. The unsupported processor issue was not expected. One of the computers on which the installation failed to install has a Xeon 3.0 GHz processor with 16 cores and an RTX 3090 video card. This computer has 64 gigs of ram, 2 terabytes of M.2 storage, and a few terabytes on conventional drives. But its processor is not supported. If your computer matches the Windows 11 requirements on paper, that doesn’t give assurance that it is actually compatible. If you want to test for yourself, the best method is to run the Windows 11 upgrade advisor.

Microsoft is using virtualization-based security (VBS) to protect processes in Windows 11. There is a sub-feature of this called Hypervisor-Protected Code Integrity (HVCI) that prevents code injection attacks. Microsoft has said that computer’s with processors that support this feature have a 99.8% crash free experience (source). For purposes of reliability and security, Microsoft has decided that this feature will be part of the baseline for Windows 11. Back in August, Microsoft made a blog post stating that they found some older processors that met their requirements and would be adding them to their compatibility list. There’s a chance that a computer that shows as no compatible today could show as compatible later.

A Windows feature that I’ve enjoyed using is Windows 2 GO. With W2G, a Windows environment is installed on a USB drive specifically made for this feature (it would not quite work on a regular drive). The W2G drive could be plugged into any PC or Intel based Mac as a boot drive. It was a great way to port an environment around to use as an emergency drive. Microsoft discontinued support of the feature some time ago. But it still worked. With Windows 11, this feature is effectively dead.

You can find both the upgrade advisor and the Windows 11 download at the following link.

https://www.microsoft.com/en-us/software-download/windows11

Windows 11 offers a a lot of consumer focused features and a new look. But my interest is in the new APIs that the OS provides. Microsoft has extended the DirectX APIs, especially in Composition and DirectDisplay. The Bluetooth APIs have extended support for low energy devices. And there is now support for haptic pen devices and more phone control.

I was already on the market for a new laptop. I’ll be getting another computer that runs Windows 11 soon enough. But in the mean time, I’ve moved one of my successful installs to my primary work area so that I can try it out as my daily driver. More to come…


Twitter: @j2inet
Instagram: @j2inet
Facebook: j2inet
YouTube: j2inet

Conferences and a Hearings, Sept-Oct 2021

During the month of October, there are a couple of developer conferences happening. Samsung is resuming what had been their regular Developers conference (there wasn’t one in 2020, for obvious reasons). Like so many other conferences, this one is going to be online on 26 October. Details of what will be in it haven’t been shared yet, but I noticed a few things from the iconography of their promotional video.

he Tizen logo is present, specifically on a representation of a TV. It looks that Samsung has abandoned the Tizen OS for anything else. They generally don’t make an announcement that they are sunsetting a technology and instead opt to quietly let it disappear. A few months ago Google made the ambiguous announcement that Samsung and Google were combining their wearable operating systems into a single platform while not directly saying that Tizen was going away. Just before the release of the Gear 4 watch (which runs Android Wear, not Tizen) Samsung made an announcement that they were still supporting Tizen. But with no new products on the horizon and the reduction in support in the store, this looks more like a phased product sunset.

Some of the other products suggested by the imagery include wearables, Smart Things (home automation), Bixby (voice assistant) and Samsung Health.

October 12-14, Google is hosting their Cloud Next conference. Registration for this conference is open now, and available at no cost. Google has made the session catalog available. The session categories include AI/Machine Learning, Application Development, Security, and more.

Sessions available at https://cloud.withgoogle.com/next

And last, if you have an interest in the USA’s developing responses to technology issues, this Thursday the Senate Committee on Commerce, Science, and Transportation is holding a hearing with Facebook’s head of safety over some recent reports published by the Wall Street Journal about the impact of it’s apps on younger audiences. The hearing (with live stream) will be Thursday, September 30, 2021 at 10:30am EDT. The livestream will be available at www.commerce.senate.gov.


Twitter: @j2inet
Instagram: @j2inet
Facebook: j2inet
YouTube: j2inet

Posts may contain products with affiliate links. When you make purchases using these links, we receive a small commission at no extra cost to you. Thank you for your support.

Sharing Resources With Your Chromebook Linux Container

If you have installed Linux on your Chromebook, you may have noticed that the file system as viewed from the files application on your Chromebook and the file system in the Linux terminal do not look alike. This is because Linux is running within a container. There are two ways to share files between your Chromebook and the Linux container.

If you open the Files application on your Chromebook, you will see an area called Linux Files. This lets you get access to files in your Linux home directory and read them from Linux doesn’t have immediate access to the files on the Chromebook. To access these, you need to explicitly share the folder with Linux. From the Files application find the folder that you want to share. Right-click on it and select “Share with Linux.” From within Linux if you navigate to the path /mnt/chromeos you will see sub-folders that are mount-points for each of the folders from Chrome that you’ve shared.

You can also share USB drives with Linux. By default, they are not available. If you open Settings and look for “Manage USB Devices” the USB drives that are connected to your machine will be listed. You can select a drive to share with Linux from there. Note that when you disconnect the drive, the next time that it is reconnected it will not automatically be shared.

The Linux container’s ports are also not exposed to your network by default. For the ports to be visible to other devices on your network, you must explicitly share them. Under settings if you look for “Port Forwarding” you will be taken to an interface where you can specify the ports that will be exposed. Note that you can only add ports in the range of 1024 to 65,535.

Hosting a Website from Home on the Pi

Under many cases, one can easily host a website from home. There are a few technical requirements that must be satisfied. But provided that they can be satisfied, making a site available on the Internet from one’s home connection isn’t hard. That said, there are some reasons one might not want to do this, such as exposing their home IP Address, or any security flaws in their at-home web server making other devices on their network vulnerable if the web server has a security flaw. There are a few ways that this can be mitigated, including ensuring that the latest security updates are installed and not exposing unnecessary services to the Internet.

Before getting started, you will want to know whether your IP address is public or behind a network address translator. This is like the difference in having your own phone number or being in a phone pool in which all of the phones are identified by the same phone number. The easiest way to figure this out is to perform a Google Search on “What is my IP address” and compare that address to the one that is reported by your router. If those addresses are the same, congrats! You’ve got a public address. If they are not the same, then your network is behind a shared IP address. Solutions for hosting in this scenario involve having traffic routed from another computer that has a public address to one’s home address. That’s out of scope of what I am trying to do here.

If you have a public IP address, you will need access to your router settings. By default, routers will not direct incoming requests to devices on your network unless they are configured to do so. The exact interface used for applying these settings are manufacturer dependent. I’ll be showing the process on an ASUS router. If you are using a different router (or even another ASUS router) the interface will look different, but the concept is the same.

For a web server, I’m going to use a Raspberry Pi. The Pi doesn’t use a lot of energy or space. I can leave it on 24-7 in a place that is out of the way. Many of the same solutions for making web applications on the desktop also run on the Raspberry Pi. I’ll be using .Net Core. But someone could also use Node or a number of other solutions. I already have the .Net Core framework and SDK installed on my Raspberry Pi. I’ll just create a default website, as I’m not concerned with the content in it for not. To do this, I made a new folder that will contain the web site and used the following command.

dotnet new web

A few moments later, the Raspberry Pi has a “Hello World!” website build. To run it, use the following command.

dotnet run

The console will output the URL to the site. It will look something like http://localhost:5000. If you open a browser on the Pi to this URL you should see the text “Hello World” render. If you use the Pi’s IP address and try this from another computer on the network, it is going to fail. But why?

By default, the website is only binding to the loopback IP address (127.0.0.1 or ::1). The site is only visible from within the computer. To change this, we could either have the site bind to a specific IP address (the Pi could have several IP addresses) or we could tell the site to bind to all network adapters on the computer. For .Net Core, we can change the address and port that the site binds to by editing Properties/launchSettings.json. Close to the bottom of the file is a setting named applicationUrl. It has a list of IP address/port combinations separated by semicolons (;). Add to this list the value http://*:5000 (feel free to use a different port number).

    "staticwebsite": {
      "commandName": "Project",
      "dotnetRunMessages": "true",
      "launchBrowser": true,
      "applicationUrl": "https://localhost:5001;http://localhost:5000;http://*:5000",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
     }

Now if you return to the other computer and navigate, in your browser, to http://<ip address>:5000 you will see the text “Hello World. The site is not visible to other computers on your network. But what about other computers on the Internet? This is where you will need to look at your router settings.

Your home router’s ip address will usually be the same as what the computer reports as its gateway address. Depending on your operating system, you can see your computer’s network settings by opening a terminal and using the command ipconfig (Windows) or by using the command ip r (linux). Example output of both commands follow.

PS C:\Users\SomeUser> ipconfig

Windows IP Configuration


Ethernet adapter Ethernet:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Ethernet adapter Ethernet 2:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::e055:c0d4:65ac:49d1%3
   IPv4 Address. . . . . . . . . . . : 192.168.50.79
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.50.1
pi@raspberrypi:~ $ ip r
default via 192.168.50.1 dev wlan0 proto dhcp src 192.168.50.50 metric 304
192.168.50.0/24 dev wlan0 proto dhcp scope link src 192.168.50.50 metric 304

Type this IP address into your browser. You will need to know the user ID and password to your router, as it will likely ask you for them. Within your router, you will want to find the port forwarding settings. On my Asus Router, I can get to this page by selecting “Wan” and then selecting the “Virtual Server/Port Forwarding” tab. My router gives me a warning that ports 80 (usually used for HTTP) and ports 20 and 21 (used for FTP) are already used by the router for some services that it provides. I don’t actually have these services turned on, thus I wouldn’t run into conflicts. Nevertheless, I will use a different port.

Asus Router Port Forwarding Page

Let’s say that I want the world to access my site through port 8081. You may recall that my Pi was hosting its site on port 5000. It is not necessary for the ports to be the same. In my interface there is a switch to enable port forwarding.Clicking on “Add Profile” will create a new entry. For this entry, I need to specify this forwarding is for TCP traffic coming in on external port 80. That traffic should be forwarded to port 5000 on the address of my Raspberry Pi, which is 192.168.50.50.

Port Forwarding Settings

After saving these settings, I turned off WiFi on my phone (to ensure it was not on the same network) and typed my home network’s IP address into my browser. I see the “Hello World” page. Great! But there is an issue here. I’m going to find it hard to remember my IP address. Also, my IP address changes every so often. Even if I keep track of it, it could change without notice. The solution to this is to use a Dynamic DNS. The router I’m using comes with this service built in. There are also options for networks where the router does not have this option.

On this Asus router, the option is under the Wan settings under a tab named DDNS. To enable the feature, I can set the option Enable the DDNS Client to Yes and select which DDNS service I want to use. If I use the default, the only setting I need to enter is the name that will be used as a portion of the domain.

Supported DDNS services for the Asus Router

For the other services, you will require additional information. I’ve used NO-IP before. I’ll use that one. Before using this service, you will want to visit no-ip.com and create a free account. After signing in, you can set your DDNS host names. You can have up to three domains setup on the free account. When setting this up, note that the IPv4 address defaults to that of the device from which this hostname is created.

DDNS Setup Screen

The actual domain used will be the concatenation of what is entered in the Hostname setting, followed by a period, followed by the domain selected from the drop down. Once you’ve saved this information, return to your router DDNS settings. Enter the complete host name, your NO-IP username, and password. After selecting “Apply” you should see a notification that the settings were successfully applied. Now, when you, or anyone else enter that domain name, your site will come up.

If your router doesn’t support a DDNS client, then you can build a DDNS updater on your Raspberry Pi. On your Pi, make a folder for the Dynamic Update Clienc (DUC). Enter that folder from a terminal and download and unpackage and build the DUC with the following commands.

wget https://www.noip.com/client/linux/noip-duc-linux.tar.gz
tar vzxf noip-duc-linux.tar.gz
cd noip-2.1.9-1/
sudo make
sudo make install

The last command, sudo make install, will ask for your NO-IP account information. You will also be asked for an update interval (in minutes). Accept the default of 30 minutes. From hereon, the Raspberry Pi will update the DDNS.


Twitter: @j2inet
Instagram: @j2inet
Facebook: j2inet
YouTube: j2inet

Posts may contain products with affiliate links. When you make purchases using these links, we receive a small commission at no extra cost to you. Thank you for your support.

Raspberry Pi Heat Sink

Pi Touch Screen with Case

Installing .Net Core and Visual Studio Code on Chromebook

Having a portable light-weight development environment is important to me. I’m always on the lookout for a solution. There have been a number of solutions I’ve found for which I’ve been optimistic, but over a longer period of time they just haven’t worked out. Samsung had Linux on Dex, which allowed certain phones to run a full Ubunto Linux environment. But they later discontinued it and removed support for it. Microsoft Windows2Go, which allowed a full Windows environment to be installed on a portable USB drive and moved from one computer to another. I still have a few of these drives and use them, but Windows has dropped support for them and replacement drives are hard to find. I’ve managed to use a Raspberry Pi as a portable development environment before too. But now I have what looks to be an improved option. Many Chromebooks support a Linux environment.

I’m using a Samsung Chromebook Plus. While the unit that I have is older, it has seen some significant changes throughout its life. When I first received it, to edit code I had to use a code editor from the Chrome store. This was nowhere near the best editor that I have used, but it worked. The Chromebook later gained the ability to run Android applications. In the Google Play Store there were some code editors, but once again, not the best. My Chromebook has had a linux environment on it for a while, and I’ve recently installed the Visual Studio Code. Finally! A **real** code editor!

Before installing .Net Core or Visual Studio Code, the Chromebook must be enabled for Linux. Not all Chromebooks support this feature. To enable this feature, open the Chromebook settings. Search for “Linux.” If your Chromebook supports Linux you will see the option “Turn on.” Press the button and wait a few moments. The Chromebook will install the components needed for Linux. After the installation is complete, you may need to restart your Chromebook. Once the Chromebook has restarted you will have the program “Terminal” on your computer. If you open this, you will be within the Linux environment. Update the environment, and install a text editor to use at the terminal.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install nano

From within the terminal, you can install dotnet core the same way that you would on a Raspberry Pi. Visit https://dotnet.microsoft.com and select “Downloads.” You will see options for downloading the .Net Framework or .Net Core. Select the “all .Net Core Downloads” link. The next screen has a list of the .Net Core downloads. At the time of this writing the most recent version is the .Net Core version 5.0. Select it to see the builds. My Chromebook uses an ARM processor, though there are some that use x86 processors. For my Chromebook I must download the ARM 32-bit Linux version. The next page will share a direct download link. Copy this link and open the terminal.

In the terminal, use the wget command to download the .Net Core installation. For the current version the command looks like the following.

wget https://download.visualstudio.microsoft.com/download/pr/97820d77-2dba-42f5-acb5-74c810112805/84c9a471b5f53d6aaa545fbeb449ad2a/dotnet-sdk-5.0.301-linux-arm.tar.gz

After a few moments, the installation has downloaded. To install it, we must make a folder into which it till be installed, unpackage the tar file, and add this folder to the path.

sudo mkdir /usr/share/dotnet
sudo tar xvf name_of_archive.tar.gz
sudo nano ~/.profile

That last command will open your .profile file, which contains a list of commands that are run when you login. Go to the end of the file and add these two lines.

export PATH=$PATH:/usr/share/dotnet
export DOTNET_ROOT=/usr/share/dotnet

.Net Core is installed, but we need to restart the Linux environment before it will work. Right-click on the terminal icon in the task bar and select the option to “Shut down Linux.” When open it again, the changes will be applied. If you find yourself opening the terminal frequently.

Installing Visual Studio Code is easier. Navigate to http://VisualStudio.com. You will see links to different editions of Visual Studio. Select the one for Visual Studio Code. Select the option to see More. On the next page select the download for the Linux deb ARM build of Visual Studio Code. After the file downloads, find it in your file system and double-click on it. An installer window will open. Select the option to Install. For me, the installation progress bar does not move for quite some time before it showed any change. After the installation is complete you can start it by typing ‘code’ in the command terminal or by finding it in your list of installed program.

Unencrypted HTTP on Android

Most network resources that I access with the Android applications that I build communicate over HTTPS. It isn’t often, but I sometimes access a resource over an unencrypted connection. This is usually the case for home automation and media control devices. One example of such an application is a Roku remote that I made for myself; the Roku accepts HTTP requests to simulate presses on the remote control. When I create something that needs to access resources over unencrypted HTTP, there’s a step I usually forget that leaves me wondering why I am receiving null responses back from my request.

When Android P was released, Google implemented a change to encourage developers to use encrypted HTTPS instead of unencrypted HTTP.

As part of a larger effort to move all network traffic away from cleartext (unencrypted HTTP) to TLS, we’re also changing the defaults for Network Security Configuration to block all cleartext traffic. You’ll now need to make connections over TLS, unless you explicitly opt-in to cleartext for specific domains.

Android Developer Blog, Dave Burke, BP of Engineering for Android

The most recent time that I forgot to enable unencrypted communication was perplexing. I was communicating with a device over UDP and HTTP, and I saw that the device was responding to the UDP requests and was a bit confused before I remembered the step that I had missed.

To enable unencrypted clear text communication generally within an application, there is an additional attribute that must be added to the <application> element in the application’s manifest.

android:usesCleartextTraffic="true"

That is going to enable HTTP traffic for all domains. I used this option for communication over a local network since there isn’t a specific domain that I can target for someone’s home connection. When there is a specific domain that your application must communicate with, you can create a network security policy permitting unencrypted communication to that domain. Before doing this, consider what information that your application is sending. If there is any personally identifying or sensitive information in your messages, then this option is not acceptable. If for some reason you cannot enable HTTPS (such as the domain being controlled by another entity) and you’ve reviewed the risks and consider them acceptable, then you can move forward with allowing unencrypted communication with that domain. To do that, you will need to create a new XML file resource in res/xml, giving it the name of your choosing. The contents of the file will look like the following. You will need to place the domain(s) of interest in the configuration.

<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
    <domain-config cleartextTrafficPermitted="true">
        <domain includeSubdomains="true">yourtargetdomain.net</domain>
    </domain-config>
</network-security-config>

In the application’s manifest, add a reference to this policy. Assuming the above file is named “network_security_config.xml” the manifest entry would look like the following.

<application android:networkSecurityConfig="@xml/network_security_config" ... />

While using SSL/HTTPS is generally preferable and lower risks, there may be times when you must fallback on unencrypted clear text communication. When this happens, for Android-P and later your application must explicitly opt-in to clear text communication.

A Cellular Connection for the Raspberry Pi

I recently setup the Simcom 7600 on a Jetson Nano. When I spoke of it I had mentioned that there is also a version of the cellular model that is specifically made for the Raspberry Pi. After a few delivery delays I have one in my hands and am looking at it now. Since both the Pi and the Jetson are Linux based ARM devices and the models are both using the same chipset my expectation is for the setup to be similar. There are two primary versions of the device. The version that I am using was released in 2020 December. There are some slight differences in the labels on the headers between these models. The version that I use is also without an SD card slot. Some of the older versions have these.

When you purchase a Simcom 7600, there is usually a letter following that number. The letter lets you know which variant that you have. Different variants are best supported by mobile networks in different regions. E-H work best in Southeast and West Asia, Europe, and Africa. A-H are primarily for North America, Australia, New Zealand, Taiwan, and Latin America.

The Simcom 7600 connects to a Pi the same way you would connect any other hat. Well, mostly. There are some options in how it is connected. The most obvious way is to connect the 4G hat to the 40-pin connector. I used this method. To do so, I had to remove the cooling fan that I had on my Pi. Included in the box are a couple of standoffs and screws for securing the 7600 to the board. Personally, I feel that a Pi that is using a mobile connection should also have its own battery. To secure the board and my batter I had to use a different set of standoffs. But I’ve got everything working (minus the cooling fan).

The Simcom 7600 for the Pi has a couple of USB ports and jumpers on it. Before powering it on, I went to the documentation to see what these were all for. Starting with the yellow header, there is a jumper already bridging PWR and 3V3. This is to set a power-on option. In this default state, the SIMCOM 7600 will turn on any time that it receives power. If the jumper is moved to bridge PWR and D6, then the SIMCOM 7600 will be off by default, but the Pi can control the power state itself. A user can also control the state through the power button on the side of the device. A third option is to remove the jumper entirely. If the jumper is removed then the only way to control the devices power state is manually using the power button.

In addition to controlling power, you now also have the option to place the device in flight mode. The control flight-mode with the pi, bridge pins D4 and Flight with a jumper. If the jumper is present then then flight mode is controllable through software.

Just behind the headphone jack are another set of jumpers. The purpose of these headers was not immediately obvious to me at first. They are not mentioned in the manual. But they show up on the schematic for the SIMCOM 7600. This header is for deciding how communication with the SIMCOM 7600 will occur.

SIMCOM 7600 Communication Jumpers

The pins that lead to the SIMCOM chip itself are the TXD 3.3V and RXD 3.3V. These lines pass through a line converter to raise the signals to the voltage level that the SIMCOM uses. If the jumpers are in their top position (connecting U_RX to TXD 3.3V and U_TX to RXD 3.3V) then communication with the SIMCOM will occur over USB (specifically USBJ1). In the middle position, communication with the SIMCOM occurs over the Raspberry PI 40 pin header on pins 8 and 10 (P_TX and P_RX). In the lowest position, the USB port connects to the Pi with there being no connection made to the SIMCOM chip.

There is a second USB port on the board. What is that for? The second USB port connects directly to the SIMCOM itself. It has USB interface pins on the chip itself. That means that there are two ways to communicate with the SIMCOM 7600 chip.

There are only a few lines on the 40-pin header that interact with the SIMCOM 7600. I could restore the heatsink and fan to my SIMCOM 7600 and still allow the Pi to communicate over USB along with a few other lines. But I prefer to have the board secured to the Pi.

Leaving the settings in their default state, I’ll be communicating with the SIMCOM 7600 over both USB and using the 40-pin header. To minimize the number of things that I could forget to do that would result in the board being non-responsive, I’m going to leave it bolted to the board though to keep it more secure.

Before setup, ensure that you’ve updated the packages on your Raspberry Pi

sudo apt-get update
sudo apt-get upgrade

Ensure that the serial port on the pi is enabled. From the Pi desktop upen the Pi menu, select “Preferences.” Then select “Raspberry Pi Configuration.” In the Interfaces tab select “Enable” next to the “Serial Port” item. If it were not enabled before, you will need to reboot after you enable it.

Shutdown your Pi and remove power from it. Attach the 4G hat to the Pi and power it back up. You should see the Power light on the Pi illuminated solid red. If the Pi detects a cellular signal, the Net light will blink. If it is solid, ensure that you have securely attached the antenna and have the SIM card in place.

Open a command terminal and type

sudo lsusb

You will see some serial devices listed. Connect the Pi and the cellular modem using the USB port that is next to the cellular antenna. Then, from the command terminal, run the lsusb command again. You should see an additional item of hardware. If you do, then the Pi has detected the modem.

Let’s get the software installed. The drivers for the modem are in a *.7z file. You will need to install a tool for unarchiving the file. You also need to have a tool for interacting with the serial port.

sudo apt-get install minicom p7zip-full

Download and unpackage the example code for the SIMCOM 7600. Along side this sample code is the driver that is needed for the Raspberry Pi.

wget https://www.waveshare.com/w/upload/2/29/SIM7600X-4G-HAT-Demo.7z
7z x SIM7600X-4G-HAT-Demo.7z -r -o/home/pi
sudo chmod 777 -R /home/pi/SIM7600X-4G-HAT-Demo

When the Pi boots up, we want it to initialize the SIMCOM board. To ensure this happens, open /etc/rc.local and add the following line.

sh /home/pi/SIM7600X-4G-HAT-Demo/Raspberry/c/sim7600_4G_hat_init

After initialization, you can start interacting with the Pi hat. As a test that it is responding, you can connect to it using the minicom utility and send some AT commands and see that is responds. You can connect to it using either port /dev/ttyUSB2 or /dev/ttyUSB3.

minicom -D /dev/ttyUSB2 -b 115200