My Samsung HoloLab Model Arrived

Last week I received an e-mail from Samsung regarding the HoloLab scans that I did at the 2018 Samsung Developer’s Conference.  Shortly after the conference, I wrote about the rig that was used to do the scan.

When the photographs were taken for the scan, three poses were requested.  The first pose was standing with your arms crossed.  The second pose was standing with both of your arms out to the side.  The third pose was allowed to be a freestyle that could be whatever you wanted (just for the fun of it).

Of these three poses, I was most interested in the second pose, because arms out to the side is the most appropriate pose to use when importing a model into software for animating.  Sadly, what arrived in my e-mail was only one of the three poses, the first one.

I’m still happy to have received the one pose that I did. The model definitely resembles me.  This is speculation on my part, but I imagine that the processing of the 52 images that make up a single scan is time consuming. Considering the large number of participants at the conference who had the scans done, receiving all three model poses may be wishful thinking.

 

HololabScan

SSDP Discovery in HTML

While implementing a few projects I decided to implement them in HTML since it would work on the broadest range of my devices of interest. My projects of interest needed to discover additional devices that are connected to my home network. I used
SSDP for discovery.

SSDPDiscovery

SSDP (Simple Service Discover Protocol ) is a UDP based protocol that is a part of UPnP for finding other devices and services on a network. It’s implemented by a number of devices including network attached storage devices, Smart TVs, and home automation systems. There are a lot of these devices that expose functionality through JSON calls. You can easily make interfaces to control these devices. However, since the standards for HTML and JavaScript don’t include a UDP interface, how to perform discovery isn’t immediately obvious. Alternatives to SSDP include having the user manually enter the IP address of the device of interest or scanning the network. The latter of those options can raise some security flags when performed on some corporate networks.

For the most part, the solution to this is platform dependent. There are various HTML based solutions that do allow you to communicate over UDP. For example, the BrightSign HTML5 Players support UDP through the use of roDatagramSocket. Chrome makes UDP communication available through chrome.udp.sockets. Web pages don’t have access to this interface (for good reason, as there is otherwise potential for this to be abused). Although web apps don’t have access, Chrome extensions do. Chrome Extensions won’t work in other browsers. But at the time of this writing Chrome accounts for 67% of the browser market share and Microsoft has announced that they will use Chromium as the foundation for their Edge browser. While this UDP socket implementation isn’t available in a wide range of browsers, it is largely available to a wide range of users since this is the browser of choice for most desktop users.

To run HTML code as an extension there are two additional elements that are needed: a manifest and a background script. The background script will create a window and load the starting HTML into it.

chrome.app.runtime.onLaunched.addListener(function() {
    chrome.app.window.create('index.html', {
        'outerBounds': {
        'width': 600,
        'height': 800
        }
    });
});

I won’t go into a lot of detail about what is in the manifest, but I will highlight its most important elements. The manifest is in JSON format. The initial scripts to be run are defined app.background.scripts. Other important elements are the permission element, without which the attempts to communicate over UDP or join a multicast group will fail and the manifest_version element. The other elements are intuitive.

        {
            "name": "SSDP Browser",
            "version": "0.1",
            "manifest_version": 2,
            "minimum_chrome_version": "27",
            "description": "Discovers SSDP devices on the network",
            "app": {
              "background": {
                "scripts": [
                  "./scripts/background.js"
                ]
              }
            },
          
            "icons": {
                "128": "./images/j2i-128.jpeg",
                "64": "./images/j2i-64.jpeg",
                "32": "./images/j2i-32.jpeg"
            },
          
            "permissions": [
              "http://*/",
              "storage",
              {
                "socket": ["udp-send-to", "udp-bind", "udp-multicast-membership"]
              }
            ]
          }    

Google already has a wrapper available as a code example chrome.udp.sockets that was published for using Multicast on a network. In it’s unaltered form the Google code sample assumes that text is encoded in the 16-bit character encoding of Unicode. SSDP uses 8-bit ASCII encoding. I’ve taken Google’s class and have made a small change to it to use ASCII instead of Unicode.

To perform the SSDP search the following steps are performed.

  1. Create a UDP port and connect it to the multicast group 239.255.255.250
  2. Send out an M-SEARCH query on port 1900
  3. wait for incoming responses originating from port 1900 on other devices
  4. Parse the response
  5. Stop listening after some time

The first item is mostly handled by the Google Multicast class. We only need to pass the port and address to it. The M-SEARCH query is a string. As for the last item, it isn’t definitive when responses will stop coming in. Some devices appear to occasionally advertise themselves to the network even if not requested. In theory you could keep getting responses. At some time I’d suggest just no longer listening. Five to ten seconds is usually more than enough time. There are variations in the M-SEARCH parameters but the following can be used to ask for all devices. There are other queries that can be used to filter for devices with specific functionality. The following is the string that I used; what is not immediately visible, is that after the last line of text there are two blank lines.

M-SEARCH * HTTP/1.1
HOST: 239.255.255.250:1900
MAN: "ssdp:discover"
MX: 3
ST: ssdp:all
USER-AGENT: Joel's SSDP Implementation
    

When a response comes in, the function that we assign to MulticastScoket.onDiagram will be called with a byte array containing the response, the IP address from which the response came, and the port number from which the response was sent (which will be 1900 for our current application). In the following code sample, I initiate a search and print the responses to the JavaScript console.

const SSDP_ADDRESS = '239.255.255.250';
const SSDP_PORT = 1900;
const SSDP_REQUEST_PAYLOAD =    "M-SEARCH * HTTP/1.1\r\n"+
                                "HOST: 239.255.255.250:1900\r\n"+
                                "MAN: \"ssdp:discover\"\r\n"+
                                "MX: 3\r\n"+
                                "ST: ssdp:all\r\n"+
                                "USER-AGENT: Joel's SSDP Implementation\r\n\r\n";

var searchSocket = null;

function beginSSDPDiscovery() { 
    if (searchSocket)
        return;
    $('.responseList').empty();
    searchSocket = new MulticastSocket({address:SSDP_ADDRESS, port:SSDP_PORT});
    searchSocket.onDiagram = function(arrayBuffer, remote_address, remote_port) {
        console.log('response from ', remote_address, " ", remote_port);
        var msg = searchSocket.arrayBufferToString8(arrayBuffer);
        console.log(msg);        
    }
    searchSocket.connect({call:function(c) {
        console.log('connect result',c);
        searchSocket.sendDiagram(SSDP_REQUEST_PAYLOAD,{call:()=>{console.log('success')}});
        setTimeout(endSSDPDiscovery, 5000);
    }});    
}

Not that parsing the response strings is difficult, by any means it would be more convenient if the response were a JSON object. I’ve made a function that will do a quick transform on the response so I can work with it like any other JSON object.

function discoveryStringToDiscoveryDictionary(str) {
    var lines = str.split('\r');
    var retVal = {}
    lines.forEach((l) => {
        var del = l.indexOf(':');
        if(del>1) {
            var key = l.substring(0,del).trim().toLowerCase();
            var value = l.substring(del+1).trim();
            retVal[key]=value;
        }
    });
    return retVal;
}    

After going through this transformation a Roku Streaming Media Player on my network returned the following response. (I’ve altered the serial number)

{
    cache-control: "max-age=3600",
    device-group.roku.com: "D1E000C778BFF26AD000",
    ext: "",
    location: "http://192.168.1.163:8060/",
    server: "Roku UPnP/1.0 Roku/9.0.0",
    st: "roku:ecp",
    usn: "uuid:roku:ecp:1XX000000000",
    wakeup: "MAC=08:05:81:17:9d:6d;Timeout=10"    ,
}

Enough code has been shared for the sample to be used, but rather than rely on the development JavaScript console,  I’ll change the sample to show the responses in the UI. To keep it simple I’ve defined the HTML structure that I will use for each result as a child element of a div element of the class palette. This element is hidden, but for each response I’ll clone the div element of the class ssdpDevice; will change some of the child members; and append it to a visible section of the page.

        
 <html>
    <head>
        <link rel="stylesheet" href="styles/style.css" />
        http://./scripts/jquery-3.3.1.min.js
        http://./scripts/MulticastSocket.js
        http://./scripts/app.js
    </head>
    <body>
Scan Network

 

</div>

address:
location:
server:
search target:

</div> </div>

</body> </html>

 

The altered function for that will now display the SSDP responses in the HTML is the following.

        function beginSSDPDiscovery() { 
            if (searchSocket)
                return;
            $('.responseList').empty();
            searchSocket = new MulticastSocket({address:SSDP_ADDRESS, port:SSDP_PORT});
            searchSocket.onDiagram = function(arrayBuffer, remote_address, remote_port) {
                console.log('response from ', remote_address, " ", remote_port);
                var msg = searchSocket.arrayBufferToString8(arrayBuffer);
                console.log(msg);
                discoveryData = discoveryStringToDiscoveryDictionary(msg);
                console.log(discoveryData);
        
                var template = $('.palette').find('.ssdpDevice').clone();
                $(template).find('.ipAddress').text(remote_address);
                $(template).find('.location').text(discoveryData.location);
                $(template).find('.server').text(discoveryData.server);
                $(template).find('.searchTarget').text(discoveryData.st)
                $('.responseList').append(template);
            }
            searchSocket.connect({call:function(c) {
                console.log('connect result',c);
                searchSocket.sendDiagram(SSDP_REQUEST_PAYLOAD,{call:()=>{console.log('success')}});
                setTimeout(endSSDPDiscovery, 5000);
            }});    
        }    

Working with non-SSL Web Services within an SSL page

I was making a Progressive Web App (PWA) and encountered a problem pretty quickly.  PWAs need to be served over SSL/HTTPS.  The services that they access must also be served over SSL (a page served over SSL cannot access non-SSL resources).  Additionally, since my app is being served from a different domain, there must be a Cross Origin Resource Sharing header permitting the application to use the data.  My problem is that I ran into a situation where I needed to access a resource that met neither of these requirements.

Failed to load http://myUrl.com: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://SomeOtherURL.com' is therefore not allowed access.

The solution to this seemed obvious: a proxy service that would consume the non-SSL feed and make the results available over HTTPS.  There exists some third party services that can do this for you (My SSL Proxy, for example).  But the services that I found were not meant for applications and generally don’t add the required CORS headers.  Implementing something like this isn’t hard, but for a lightweight application for which I wasn’t planning on making any immediate revenue, I wanted to minimize my hosting costs.  This is where two services that Google provides come into play.

The first Google service is Firebase.

Firebase (available at https://Firebase.Google.com) allows you to host static assets in the Google cloud.  These assets are servers over SSL.  This was a perfect place for hosting most of the source code that was going to run on the mobile device.

As for the service proxy, I made a proxy service that ran on the second Google service: App Engine.  Google’s cloud service App Engine (available at https://cloud.google.com/appengine/) allowed me to write my proxy service using NodeJS (available at https://nodejs.org/).  I had it query the data I needed from the non-SSL service and cache the data for 30 seconds at a time.  All of Google’s services use SSL by default, so I didn’t have to do anything special.  When returning the response I added a few headers to handle CORS requirements.  Here’s the code for the node server.  If you use it, you will need to modify it so that any parameters that you need to pass to the non-SSL service are passed through.

const http = require('http')
const port = 80;
const MAX_SCHEDULE_AGE = 30;
const SERVICE_URL=`YOUR_SERVICE_URL`

var schedule = '[]';
var lastUpdate = new Date(1,1,1);


function timeDifference(a,b) { 
    var c = (b.getTime() - a.getTime())/1000;
    return c;
}

function sendSchedule(resp) {
    resp.setHeader('Access-Control-Allow-Origin', '*');
	resp.setHeader('Access-Control-Request-Method', '*');
	resp.setHeader('Access-Control-Allow-Methods', 'OPTIONS, GET');
	resp.setHeader('Access-Control-Allow-Headers', '*');
    resp.end(schedule);
}
const requestHandler = (request,response) => {
    var now = new Date();
    var diff = timeDifference(lastUpdate, now);
    if(diff>MAX_SCHEDULE_AGE) {
        console.log('schedule is stale. updating');
        sendSchedule(response);
        return;
    }
    updateSchedule((d)=> {
        console.log('schedule updated')
        sendSchedule(response);
    });
    console.log(request.url);

}

const server = http.createServer(requestHandler);

const https = require('https');


function updateSchedule(onUpdate) { 
    https.get(SERVICE_URL, (resp) => {
        let data = '';
        resp.on('data', (chunk) => {
            data += chunk;
        });
        resp.on('end', ()=> {
            schedule = data;
            lastUpdate = new Date();
            if(onUpdate) {
                onUpdate(schedule);
            }
        })
    });
}

server.listen(port, (err) => {

    if(err) {
        return console.log('something bad happened');
    }
    console.log(`server is listening on port ${port}`);
    updateSchedule();
}) 

One of the other advantages of having this proxy service is that there is a now a layer for hiding any additional information that is necessary for accessing the service of interest.  For example, if you are communicating with a service that requires some key or app id for access, that information would never flow through to the client.

Some configuration was necessary for deployment, but not much.  I had to add a simple app.yaml file to the project.  These are the contents.

# [START runtime]
runtime: nodejs10
# [END runtime]

Deployment of the application was unexpectedly easy.  I already had the source code stored in a git repository.  App Engine exposes a Linux terminal through the browser.  I cloned my repository and typed a few commands.

$  export PORT=8080 && npm install
$  gcloud app create
$  gcloud app deploy

After answering YES to a configuration prompt, the application was deployed and running.

One might wonder why I have the code for my application hosted in two different services.  I could have placed the entire thing in App Engine.  My motivation for separating them is that I plan to have some other applications interface with the same service.  So I wanted to keep the code (for specific clients of the code) separate from the service interface.

Augmented Reality with Samsung XR SDK

Samsung showed the XR SDK at the 2018 Developers Conference. While Microsoft has generally presented their reality technologies as being along a spectrum (ranging from completely enveloping the user to only placing overlays on the real world) it has always been something that has involved a head mounted device. Samsung presents AR as something that is either viewed through a head mounted device or something that a person views through a portable hand held window into another world.  The language used by various companies varies a bit. Microsoft calls the their range of technologies “mixed reality.” Samsung calls theirs SXR which stands for Samsung Extended Reality.

It was several years ago that Samsung first showed it’s take on VR with the release of the Note 4 and the developer’s edition GearVR. The GearVR is now available as a consumer product, but Samsung took an economical approach to initial hardware for head mounted augmented reality. Instead of creating custom hardware they took some off the shelf products and mixed them together to make an economical headset.

Samsung AR Headset
Experimental AR Headset using off the shelf parts
Part Description Cost Source
AR Headset 90° FOV “Drop-In” phones 4.5 inches to 5.5 inches, 180g 65.99 USD
External Camera ELP VGA USB camera module with 100° FOV lens 24.69 USD
OTG connector Wavlink USB 3.1 Type C Male to USB 3.0 Type A Female OTG Data Connector Cable Adapter 5.99 USD Amazon
Total Cost 95.USD

The Samsung XR SDK is almost a super set of the the GearVR SDK. I say “almost” because with a proper super set you would find all the same class names that you would expect from the GearVR SDK. In the Samsung XR SDK the classes exists within a new namespace and have been renamed. GearVR programs could be ported over with some changes to the class names being invoked.

In development is an API standard for AR/VR experiences named OpenXR. Once the standard is defined and released Samsung plans for their XR SDK to be an implementation of this standard.

While the GearVR SDK was specifically for Samsung devices and the Samsung headset the Samsung XR SDK will run on non-Samsung devices for through-the-window AR but will run on the Oculus GO and Samsung devices for stereoscopic experiences.

 

Linux On Dex: Works on WiFi Tab S4 Models Only

Update 2018-Dec-11: I’ve spoken to a LoD team member and to jump straight to the point of you have a LTE Tab S4 then simply put the required update isn’t available at this time and there is no information on when it will be available.

Some people trying to install Linux on Dex are running into an obstacle. After installing he app and trying to run it they get the following error message.

Linux on Dex requires your device to have the latest software o support some features.

After this message is acknowledge the application closes. If someone with this error checks for updates in the app store or for updates to the operating system they get notification that everything is up to date. What’s going on? I contacted LoD support about this and got back the following response.

Currently, the Linux on DeX(beta) requires latest SW for Galaxy Note9 and Galaxy Tab S4. SW update schedule may vary depends on the region and carrier.

Currently, the Linux on DeX(beta) requires latest SW for Galaxy Note9 and Galaxy Tab S4.
SW update schedule may vary depends on the region and carrier.

What does this mean? It means that your device doesn’t have a update that is required for DeX and that your carrier might not have released it.  Devices sold through a carrier can be a bit slower in receiving their updates. Samsung hasn’t been specific on the updated needed.  I’ve communicated with someone on the Linux on Dex team and was told that LTE tablets in general do not have the update that is required for Linux on Dex. Additionally the person told me that there is no information available on when particular updates will work their way through certain carriers.

BTW: Unlocking your device and installing a SIM from another carrier will not change this; this behaviour is dependent on the carrier for which the device was made, not on the SIM that happens to be in the device at the time.

Samsung Announced Exynos 9 with NPU

 

Consistent with what they said at the developer’s conference about wanting to extend the reach of their A.I. Samsung has announced a new System on Chip (SoC) with some A.I. related features. The Exynos 9 Series 9820 processor. The processor contains an NPU, a unit for processing neural networks at speeds faster than what could be done with a general purpose processor alone. The presence of this unit on the device hardware makes possible device side experiences that would have previously required that data be sent to a server for processing. This may also translate into improvements in AR and VR experiences.

The NPU isn’t the only upgrade that comes with the processor. Samsung says the 9820’s new fourth generation custom core delivers a 20% improvement in single core performance or 40% in power efficiency compared to is predecessor. Multicore performance is said to be increased around 15%. The Exynos 9820 also has a video encoder capable of decoding 4K video at up to 150 frames per second in 10-bit color. The processor goes into mass production at the end of this year.

Source: Samsung

Bixby Developer Studio

Samsung says they would like to have AI implemented in all of their products by 2020. From the visual display shown during the SDC 2018 conference it appears their usage of “all” is intended to be widely encompassing. Phones, car audio systems, refrigerators, air conditioners…

Samsung is inviting developers to start engaging in development for their conversational AI. Now they have made the same tools that they use for Bixby development internally available publically. The development portal and the development tools for Windows and OS X are available now at: https://bixbydevelopers.com

Galaxy Home, a Bixby enabled smart speaker, was showcased as a target implementation for the SDK.

The “Media Control API” will be available to content partners this December for adding deeper control into applications. Samsung says Netflix and Hulu are on board and will begin development with it next year.

The Samsung Frame TVs are also being opened to developers by way of the Ambient Mode SDK. This will allow developer content to show when the TV is in it’s standby mode.

Samsung HOLOLAB

1T3A0114.JPG

One of the technology implementations displayed at SDC 2018 was their HOLOLAB. Curious as to what it is I walked over. It was a rig for performing 3D scanning of subjects. A few of the tangible outcomes included some pretty elastic 3D printed statues of people, a volume metric display showing a subject, and a display showing one of the scanned subjects dancing.

I was more curious about the implementation. I got a picture peering into the space to see how the physical structure was built. Inside the space the ambient lighting was ramped up by way of LEDs distributed around the space and 52 Point Grey Research cameras, all mounted to various places on an 8020 rig.

The software is custom built. I asked of its commercial availability and the Samsung rep I spoke to told me, “it was a work in progress.” The rep indicated that they are planning to have some popup locations available in the future for more people to try it and receive a copy of their models.

It is still early in the conference, but I might take a moment to try it out myself.

1T3A0122.JPG
Inside the HoloLab

1T3A0115.JPG

 

Tizen 5.0 Released

Tizen-Pinwheel-On-Light-RGB

Tizen 5.0 was released a few days ago. This is a week in advance of the Samsung Developer’s Conference 2018. For those keeping count until now the most recent version of Tizen prior to now (4.0) could be found on the Galaxy Watch and Samsung TVs. There are Mobile devices that run Tizen. But I am in the USA and those devices are not sold here (so I won’t speak on them much).

What’s new in Tizen 5.0?  There is improved IoT support, support for Bixby, and support for glTF. glTF is a format that aims to provide efficient loading of 3D scenes, but it is being added to Tizen with intent for it to be applied to watch faces.  With the upgrade we are also getting improved debugging support and a new version of Tizen.Net.  It also looks that they are deprecating the UI Builder tool in Tizen.

I get the feeling that I’ll hear more about this update when I go to the conference next week. I’ll post more from the conference as I find out 🙂

  • C# API for Display Control
  • Native API for Multi-LED control
  • Compressed File System
  • Low Memory Management
  • Upgrade to source libraries
    • icu (60.2)
    • sqlite (3.24.0)
    • json-glib (1.4.2)
    • wayland (1.15.0)
    • efl (1.21)
  • Updated Watchface Complication Framework
  • App Control API
  • Minicontrol API
  • WebView Control added
  • Network Firewall
  • Bluetooth 5.0
  • Neural Network Runtime

Samsung Developer Conference 2018 Registration Open

I previously mentioned that the dates for the Samsung Developer’s conference were announced. Registration is now open.  If you register before September 12 you can register for the lowest available price. The registration form is available over at https://www.samsungdeveloperconference.com/ . Registration is also possible on site. Currently Registration is 299 USD (+tax). After the 12th it will go up to 399 USD. On site registration is 499 USD.

Registration for the Samsung Developer's Conference, 2018

Obtaining the Connection String for a Provisioned Windows IOT Device

Playing with the code that I was using to get data from my car and stream it to the cloud I did something that I knew was a no-no; I hard coded the connection string in the code. There’s a number of reasons to not do this*; it’s less secure as someone can potentially extract the connection string and use it for unauthorized access and if the connection string ever needs to change then code needs to be recompiled and redeployed.

When a Windows IOT device is provisioned there is a connection string that is managed by the device; your application can take advantage of this and need not worry about the details of how it is stored. To make use of this there are a few libraries that you need to add to your UWP project. These include the followings.

  • Microsoft.Azure.Devices
  • Microsoft.Azure.Devices.Client
  • Microsoft.Devices.Tpm

With the classes in these libraries you can obtain the ID of the device and then use that ID to request an Azure DeviceClient class that is initialized with the connection string that the device is managing.

Here’s the code to do this.

DeviceClient _deviceClient;

void InitClient()
{            
    TpmDevice tpm = new TpmDevice(0);
    string hostName = tpm.GetHostName();
    string deviceId = tpm.GetDeviceId();
    string sasToken = tpm.GetSASToken();            
    var client = DeviceClient.Create(
        hostName, 
        AuthenticationMethodFactory.CreateAuthenticationWithToken(deviceId, sasToken),
        TransportType.Mqtt
    );
}

Connecting to Bluetooth RFCOMM with UWP

Soon I will be finished with a project for streaming engine data from a car and into an Azure Data Lake for analysis. I plan to post the full code for this project. But before I do I wanted to talk just about making the Bluetooth connection. The last time I wrote on this procedure it was for a Windows phone 8 project. While part of the UWP ancestry the method of connection in that project would not work in a UWP application today. Things have changed since then and the code that I used there for establishing a connection would not work today.  Treat the code used in this project as psuedo-code for now. While it was copied from a working project I’ve done some modifications to simplify it and focus on a few things here.

Getting information from the engine is easy. Every car sold in the USA (and many  parts of the world) have a diagnostic connector that can be used to query various data from the car. Getting this data is just a matter of having an interface that uses what ever protocol (there’s more than one) that your car happens to use. Don’t worry, it’s not necessary to figure this out yourself. There are products that will abstract this away and provide a common interface and protocol that can be used to communicate with all of these different protocols. One of the most popular solutions is based on a chip-set known as the ELM327. Solutions based on the ELM327 (or an ELM327 clone) will give you a serial interface through which you can send AT commands and other queries to get this information. The products that I am using use a Bluetooth interface for communicating with a computing device. You can also find implementations that use WiFi and RS232.

For development and testing there’s going to be a lot of time spent with a computer in or near a car (unless you also go out and purchase an OBD emulator). To minimize the amount of time that you have to spend around a car you may want to power one of the adapters outside of a car. I did this using a battery (sometimes a power supply) and and an OBD wire harness. Pin number 4 on the harness needs to be connected to ground and pin 16 to positive. A voltage range of 9 to 16 volts should work fine. With the device connected to power you can query information about the device itself but nothing about the car.  While this doesn’t sound very useful at first it’s enough to get the basic Bluetooth connectivity code working.

OBDSetup

For development hardware I am using a Dragonboard 410x running Windows IOT. The UWP based solution, while targeting Windows IOT, runs just fine from a PC. I believe it will run fine from a Raspberry Pi 3 also. Just for fun I tried an old Windows 10 device that I still have in the house and the code ran without any modifications needed from there. So grab what ever Windows 10 device that you have and try it out!

For the application to be able to connect to the device over Bluetooth there are some capabilities that must be declared in the UWP application. I don’t suggest using the Visual Studio for doing this. It doesn’t expose all of the necessary capability settings that are needed. It’s easier to do this in the text editor.

Right-clicking on the project’s manifest gives the option to open it in the code editor. Scrolling down to the Capabilities section I added the following. Some of these properties are needed to access the Bluetooth serial port. Some others are there because they are parts of the project that I will talk about in future post.

  
  <Capabilities>
    <apability Name="internetClient" />
    <DeviceCapability Name="bluetooth" />
    <DeviceCapability Name="proximity" />
    <DeviceCapability Name="location" />
    <DeviceCapability Name="bluetooth.rfcomm">
      <Device Id="any">
        <Function Type="name:serialPort" />
      </Device>
    </DeviceCapability>
  </Capabilities>

I had already paired the Bluetooth device with my Windows IOT device. While the devices are paired with each other the application doesn’t know about the pairing. It still needs to discover the device. To do this I use a device watcher to scan for devices. I know the one that I want to find is named “OBD”. I’ve hard coded this into my solution. The end solution that I’m making doesn’t have a UI, If it did then I would have also made use of the UI that allows a user to select a device being paired.

DataReader _receiver;
DataWriter _transmitter;
StreamSocket _stream;
DeviceWatcher _deviceWatcher;

string[] requestedProperties = new string[] { "System.Devices.Aep.DeviceAddress", "System.Devices.Aep.IsConnected" };
_deviceWatcher = DeviceInformation.CreateWatcher("(System.Devices.Aep.ProtocolId:=\"{e0cbf06c-cd8b-4647-bb8a-263b43f0f974}\")",
                                               requestedProperties,
                                               DeviceInformationKind.AssociationEndpoint);
_deviceWatcher.Stopped += (sender,x)=> {
    _isScanning = false;
    Log("Device Scan Halted");
};
EngineDataList.Add("started");
_deviceWatcher.Added += async (sender, devInfo) =>
{
    Log($"Found device {devInfo.Name}");
    System.Diagnostics.Debug.WriteLine(devInfo.Name + "|" + devInfo.Kind + "|" + devInfo.Pairing);
    if (devInfo.Name.Equals("OBDII"))
    {
     // More Code Goes Here
     //Lets talk about that in a sec
    }
};

_deviceWatcher.Start()

The above code will give result in the DeviceInfo object for the Bluetooth adapter being returned. Now we can connect to it and open up streams for reading and writing. The code for creating these streams would be where I have the comment placed in the above. The following is the code that would go in that place.

try
{
    DeviceAccessStatus accessStatus = DeviceAccessInformation.CreateFromId(devInfo.Id).CurrentStatus;
    if (accessStatus == DeviceAccessStatus.DeniedByUser)
    {
        Debug.WriteLine("This app does not have access to connect to the remote device (please grant access in Settings > Privacy > Other Devices");
        return;
    }
    var device = await BluetoothDevice.FromIdAsync(devInfo.Id);
    Debug.WriteLine(device.ClassOfDevice);

    var services = await device.GetRfcommServicesAsync();
    if (services.Services.Count > 0)
    {
        Log("Connecting to device stream");
        var service = services.Services[0];
        _stream = new StreamSocket();
        await _stream.ConnectAsync(service.ConnectionHostName,
        service.ConnectionServiceName);
        _receiver = new DataReader(_stream.InputStream);
        _transmitter = new DataWriter(_stream.OutputStream);
        
        ReceiveLoop();
        QueryLoop();
        _deviceWatcher.Stop();
    }
}catch(Exception exc)
{
    Log(exc.Message);
}
catch
{
    Log("Native Error");
}

After this runs the devices are connected to each other. The devices would interact by the Win IOT device sending a query and getting a response. For now the only query that will be sent is the string “AT\n”. On devices that accept AT command sets the command AT should only result in the response OK. This is useful for testing the connection without actually having the device perform an operation that would change the state of the system being used.

void QueryLoop()
{
    Task t = Task.Run(async () =>
    {
        
        while (true)
        {
            _transmitter.WriteString(msg);
            _transmitter.WriteString("\r");
            await _transmitter.StoreAsync();
            await Task.Delay(50);
        }
    }
    );
}

void ReceiveLoop()
{
    Task t = Task.Run(async () => {
        while (true)
        {
            uint buf;
            buf = await _receiver.LoadAsync(1);
            if (_receiver.UnconsumedBufferLength > 0)
            {
                string s = _receiver.ReadString(1);
                receiveBuffer.Append(s);
                if (s.Equals("\n")||s.Equals("\r"))
                {
                    try
                    {
                      Debug.WriteLine("Message Received:"+receiveBuffer.ToString());
                        receiveBuffer.Clear();
                    }
                    catch(Exception exc)
                    {
                        Log(exc.Message);
                     
                    }
                }
            }else
            {
                await Task.Delay(TimeSpan.FromSeconds(0));
            }
        }
    });
}

When the above runs the debug output stream will print a series of OK responses. To do something more useful with it we need to know what commands to send to it and how to parse the responses. If you send the command 01 0C while the adapter is connected to a car you will get back a hex response that starts with 41 0C followed by some additional hex numbers. The additional hex numbers are the RPMs of the car. That’s one of the metrics I started off querying because it’s something that I can change without driving the car; for this I can pull the car out of the garage, put it in the driveway, and rev the engine without worrying about carbon monoxide poisoning. I’ll talk about how to find out what metrics that you can query in your car in my next post on this project.

ELM327 Bluetooth Adapter on Amazon

Simplified UWP View Mode Base

If you’ve implemented a ViewModelBase (or some object that implements INotifyPropertyChanged) in UWP there’s a pattern that you will recognize. When a property value is being set you might check to see if the new value is different from the previous value, if it is then you assign the value to the property and call the OnPropertyChanged event. Such code might look something like this.

public abstract class ViewModelBase: INotifyPropertyChanged {
   public event PropertyChangedEventHandler PropertyChanged;
   protected OnPropertyChanged(String propertyName)
   {
      PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
   }
}

public class Employee : ViewModelBase{
   private int _employeeID; 
   public int EmployeeID
   {
      get { return _employeeID; }
      set 
      {
         if(_employeeID != value )
         {
            _employeeID = value;
            OnPropertyChanged("EmployeeID")
         }
      }
   }
}

While it works it could be better. The first improvement is in the implementation of OnPropertyChanged. If someone my a typo in the name of the property it might not noticed until runtime debugging (if at all). It would be better if such an error were caught at compile time. There’s a different implementation of OnPropertyChanged that can do this.

protected void OnPropertyChanged(Expression expression)
{
   OnPropertyChanged(((MemberExpression)expression.Body).Member.Name);        
}

With this implementation instead of typing the name of the property we can type an expression for the property. If we make a mistake in the expression it will cause a compile time error. Out Employee class will now look like the following.

public class Employee : ViewModelBase{
   private int _employeeID; 
   public int EmployeeID
   {
      get { return _employeeID; }
      set 
      {
         if(_employeeID != value )
         {
            _employeeID = value;
            OnPropertyChanged(()=>EmployeeID)
         }
      }
   }
}

That’s improved, but there’s still room for improvement. Checking for the value to be different and then conditionally assigning the value can be modularlized so that we don’t have to keep typing it. To do this I add a couple of additional methods to the ViewModelBase.

protected bool SetValueIfChanged(Expression<Func> propertyExpression, Expression<Func> fieldExpression, object value)
{
    var property = (PropertyInfo)((MemberExpression)propertyExpression.Body).Member;
    var field = (FieldInfo)((MemberExpression)fieldExpression.Body).Member;
    return SetValueIfChanged(property,field, value);
}

protected bool SetValueIfChanged(PropertyInfo pi,FieldInfo fi, object value)
{
    var currentValue = pi.GetValue(this);
    if ((currentValue == null && value == null)||(currentValue!=null &&  currentValue.Equals(value)))
        return false;
    fi.SetValue(this, value);
    OnPropertyChanged(pi.Name);
    return true;
}

The second version of this method is probably easier to understand. It uses classes from the reflection namespace to interact with the object. This allows it to operate on on any properties or fields as long as the reflection data is passed. The first implementation of this method takes an expression and gets the reflection data from it. Applying this to the Employee class it now looks like the following.

public class Employee : ViewModelBase{
   private int _employeeID; 
   public int EmployeeID
   {
      get { return _employeeID; }
      set { SetValueIfChanged(()=>EmployeeID, ()=>_employeeID, value); }
   }
}

Much better. Now if I want to reduce my typing even more instead of naming the method SetValueIfChanged I could name it something much shorter. One must still define the the fields that backup these properties. Rather than typing them out a code snippet could be defined for this pattern. A code snippet is like a template that is associated with a keystroke. I’ll talk about that in another post.

Samsung Developer Conference 2018 Dates

sdc2017logo

The dates for the 2018 Samsung Developer Conference have been announced. Outside of the dates and the location (which has been the same over the past several years) there’s not any additional information available. But if you think you might go now is the time to mark your calendars.

It’s November 7 and 8. That is a Wednesday and Thursday. As per usual it is going to be in the Moscone conference center (the West conference center).

Graphing with the HTML Canvas

Among other places I went to Peru and saw Machu Picchu! The trip back home though was long and boring.  There’s lots of ways that one can keep their mind when busy when in a non-stimulating situation though. Reading, Sudoku , cross word puzzles, so on. Sometimes I like playing with code during these times. I had started off reading a math heave book though. I generally have a notebook with me, and my notebooks are generally graph ruled. But graphing polynomials can be tedious. But hey, I had a Chromebook  with me, plenty of battery life, and no Internet access. So I decided to see what I could do with the HTML canvas. I cam up with something that worked and was useful. It’s not something that I would package and call a library by any means. After all, this is something that was put together during a layover in an airport. But nonetheless I thought it to be something that is worthy of sharing.

Drawing with the Canvas

Let’s review how to draw with the HTML5 canvas. First a <canvas> element would need to be declared within a page. Within the page’s script we grab a reference to the canvas and get the 2d context and hold onto it; the context object will be the object used for most of our interaction with the canvas. Here I take a canvas
and draw a red rectangle in it.
<canvas id="myCanvas" width="320" height="200"></canvas>

    var canvas = document.getElementById('myCanvas');
    var ctx = canvas.getContext('2d');
    ctx.fillStyle="#FF0000";
	ctx.fillRect(10,10,100,100);

Cool, we can draw on the surface. But before graphing anything we need to be able to map the data that we are graphing to the coordinate space for the graph. That’s going to be a linear transformation. Linear transformations are easy but something that I’m sure I’ll want to do several times. So I’ve made a class to figure out the the transformation for me.

Linear Transformation

Given a pair of starting and ending values this class is able to convert from on value range to another. The class has two methods. the map() method will do the conversion and rmap will perform a reverse conversion. Given the the freezing and boiling points of water in Farenheit and Celcius it could perform conversions back and forth between the two.

function linearMapping(iMin, iMax, oMin, oMax) {
	this.slope =  (oMax - oMin) / (iMax - iMin);
	this.base = oMin - this.slope*iMin;
	this.map = function(x) {
		if(typeof x != 'number')
			return null;	
		return x*this.slope+this.base;
	}
	this.rmap = function(x) {
		if(typeof x != 'number')
			return null;
		return (x - this.base)/this.slope;
	}
}

I sat for a while and I thought about the parameters that I would want to pass when creating a graph. I first envisioned passing this information as individual parameters to a function. But I didn’t have to think for long before I realized this would be a lot of parameters most of which I probably would not feel like specifying. I took a slightly different approach and decided to instead pass a parameter object. The object would be initialized with a set of workable default values that could be changed as the graph needed to be customized.

Graphing Options

Some of the basic information needed is the the size of the graph within the HTML page and the ranges of the X and Y values being displayed. I allow these values to be optionally passed in when constructing the object. I’ve also defined a color palette to be used for the graph.

function graphOptions(width, height, minX, maxX, minY, maxY) {
	this.width = width || 300;
	this.height = height || 300;
	this.minX = minX || -10;
	this.maxX = maxX ||  10;
	this.minY = minY || -10;
	this.maxY = maxY ||  10;
	this.stepX = 2;
	this.stepY = 2;
	this.backgroundColor = '#F0F0F0';
	this.lineColor = '#C0C0FF'
	this.minorLineWidth = 1;
	this.majorLineWidth = 4;
	this.dataColors = [
		"#000000",
		"#ff0000",
		"#00ff00",
		"#0000FF",
		"#FF8000",
		"#ff0080",
		"#80FF00",
		"#8000FF"
	]
}

With that we are ready to start creating our graph. We need a way of specifying where in the page the graph should be generated. I’ve made a method named makeGraph() that will accept two arguments; the parent element for the graph and the parameter object for the graph options. If for some reason the parent element isn’t specified the function will just append the graph to the page’s DOM.

Creating the Graph

This class will take care of creating the canvas object. The canvas’s context will be retained along with the options and packaged in an object as both will be needed later for plotting graphs. When the graph is created I render the graph lines on it. The canvas object is added to the page

function makeGraph(parentElement, options) {
	options = options || new graphOptions();
	var graphElement = document.createElement('canvas');

	graphElement.width = options.width;
	graphElement.height = options.height;

	var ctx = graphElement.getContext('2d');
	var newGraph = new graph(ctx, options);
	ctx.fillStyle=options.backgroundColor;
	ctx.fillRect(0,0,options.width, options.height);

	ctx.strokeStyle = options.lineColor;
	ctx.beginPath();
	ctx.lineWidth = options.majorLineWidth;
	var xOrigin = newGraph.xMapping.map(0);
	var yOrigin = newGraph.yMapping.map(0);
	ctx.moveTo(xOrigin,0);
	ctx.lineTo(xOrigin, options.height);
	ctx.moveTo(0, yOrigin);
	ctx.lineTo(options.width, yOrigin)
	ctx.stroke();
	ctx.strokeStyle = options.lineColor;
	ctx.beginPath();
	for(var i = options.minX; i<options.maxX;i+=options.stepX) {
		var xPos = newGraph.xMapping.map(i);
		ctx.moveTo(xPos,0);
		ctx.lineTo(xPos,options.height);
		ctx.lineWidth = options.minorLineWidth;
	}
	for(var i = options.minY; i<options.maxY;i+=options.stepY) {
		var yPos = newGraph.yMapping.map(i);
		ctx.moveTo(0, yPos);
		ctx.lineTo(options.width, yPos);
		ctx.lineWidth = options.minorLineWidth;
	}
	ctx.stroke();

	if(parentElement != null)
		parentElement.appendChild(graphElement)
	else
		document.body.appendChild(graphElement)
	return newGraph;
}

The result of the above is the return of a graph object that hasn’t been defined here yet. The graph object packages together the context for rendering, the graph objects, and the objects for mapping the values being represented to canvas coordinates. Something that may look odd at first is that for the Y-mapping I placed a maximum value in a minimum parameter and vice versa. This is because the coordinate system that many of us use when thinking about graphs is reversed along the Y-axis than the canvas coordinates. The general way that people think about graphs is that numbers of greater value will appear higher up on the graph. But in the canvas coordinate space the highest position has a Y-coordinate of zero and as the number increases the position maps to a position further down on a page. There’s more than one way to address this, but since the linear transfor object can already handle this if I
specify the values in a certain order that’s the solution that I used.

The function made for public use on this class ia named plot. It accepts a list of functions to be graphed. While I expect an array of functions to be passed to it if a singular function were passed that’s converted to an array of one function so that it can be treated the same way. The plot function interates through the functions passed through it passing each one to another function that does the actual work. A different color index is passed for each function.

The real work is done in plotFunction. First the X-values that fall within the range of the limits of the graph are passed to the function being mapped and the canvas-mapped result is saved to an array. The result for each X-value could either be a number or a non-number. Non-numbers will not be represented in the graph. This allows for the generation of graphs for which there may be X-values that are not part of the functions domain. If an exception occurs when calling the function being graphed the X-value association with that exception is treated as a value for which the function returns nothing. After this first pass we have an array of the canvas-mapped output values from the function.

Rendering the Graph

Next we perform the acual rendering of the output onto the graph. The function will scan ahead in the array of results to the first numerical value. This first numerical value may or may not be in the first position of the array. Once a value is found it is used as the first coordinate for a line segment. Each numerical value in the array that follows this is added to the line segment. This continues until either the end of the array is reached or a non-numerical value is encountered. In either case the accumulated points for the line segment are stroked ending. If there are more values within the array the process is repeated until the end of the array is encountered.

function graph(context, options) {
	this.ctx = context;
	this.options = options;
	this.xMapping = new linearMapping(options.minX, options.maxX, 0, options.width);
	this.yMapping = new linearMapping(options.minY, options.maxY, options.height, 0);

	this.plot = function(sourceFunctions) {
		if(!Array.isArray(sourceFunctions))
			sourceFunctions = [sourceFunctions];
		var colorNumber = 0;
		sourceFunctions.forEach((x)=> {
			this.plotFunction(x,colorNumber);
			++colorNumber;
		})
	}

	this.plotFunction = function (plotFunction, colorNumber) {
		colorNumber = colorNumber || 0;
		colorNumber = colorNumber % this.options.dataColors.length;
		var values = new Array(this.options.width);
		for(var xPos=0;xPos<this.options.width;++xPos) {
			var y = null;
			var x= this.xMapping.rmap(xPos);
			try {
				values[xPos] = this.yMapping.map(plotFunction(x))
			} catch(exc) {
				values[xPos] = null;
			}
		}
		//Find the first value that we can map
		var xPos = 0;
		while((typeof values[xPos] != 'number')&&(!Array.isArray(values[xPos]))&&(xPos < values.length))
			++xPos;
		if(xPos == values.length)
			return;
		
		while(xPos<values.length) {
			this.ctx.beginPath();
			this.ctx.strokeStyle = this.options.dataColors[colorNumber];
			this.ctx.moveTo(xPos, values[xPos]);
			while(xPos+1<values.length && typeof values[xPos+1] == 'number') {
				++xPos;
				this.ctx.lineTo(xPos, values[xPos]);
			}
			++xPos;
			this.ctx.stroke();
			while((typeof values[xPos] != 'number')&&(xPos < values.length))
				++xPos;
		}
	}
}

How Does it Look?

The graphing class is complete. Showing a graph is now only a matter of including the JavaScript in a page and using it. The simplest example of using it would be the following.

var g = makeGraph(randomPlotsArea, options);
g.plot((x)=>{return Math.sin(x); });

Here’s the result!

I’ve made a page with a few place holders for graphs. Among other things it contains the following.

<p id="randomPlots">
	
Random plots
</p> <p id="trajectoryPlotArea" >
Plot of a trajectory height for an object thrown up at 10 m/s near the earth's surface.
</p>

To render the graphs within their appropriate places I acquire a reference to the parent element in which the graph will be contained and I pass that to the makeGraph() function. Here I render the SIN function, the COSINE function (with no values returned from 2.0 to 3.0), and a x-squared

var randomPlotsArea = document.getElementById('randomPlots');				
var options = new graphOptions();
var g = makeGraph(randomPlotsArea, options);
g.plot([
	function(x){return 5*Math.sin(x);},
	(x)=>{
			if(Math.floor(x)!=2)
				return 6 * Math.cos(x);
			return null;
		},
	(x)=>{return x * x; }
	]);

Here is the result. The range for which no value is returned is apparent from the area in which the red line on the graph is not rendered.

 

Where to From Here?

Awesome! The code works! But now what?  Well, nothing for now. This was something I wrote with temporary intentions and to keep myself from being bored. That’s not to say that I’m giving up on developing a graphing library. Once back home I checked online to see what types of other graphing libraries are available. There’s a number of them, each having their own strengths. I have a direction in which I’d like to take this that is different from the others that are out there. I may revisit this, but only after a lot more thought of what I want to do, how I want this to work,  and how the variations on graphs can be specified.