|
|
|
|
11/18/2002 casey chesnut
[an error occurred while processing this directive]
I lied. /tabletStrator was supposed to be the last article before bNb went offline from me moving. But there was some leftover Tablet PC shrapnel that had been bugging me. 1) How to do Ink in Internet Explorer, which has been a recurring question in the newsgroups 2) What extra pen stroke info could be gathered from Tablet PCs to use in a signature biometric 3) /tabletStrator revisited, etc...
The problem is that the Tablet SDK Ink objects (InkCollector, InkOverlay) and controls (InkEdit, InkPicture) target WinForm apps and not WebForms. And IE has no built-in capabilities for working with ink. So on the Tablet you can use non-ink-optimized web and windows applications with only a pen ... but it is kind of a pain. Meaning you have to select the textbox, then go down to the Tablet PC Input Panel and either peck it out on a virtual keyboard, or sign down below the app and have it populate the textbox upon recognition; possibly speak it as well. It does not automagically let you do 'ink-in-place', as others have put it, directly on the application. NOTE: there is some trick to do 'Write Anywhere' which does make for a better user experience. The SDK does allow you to upgrade your existing WinForms apps to let them directly support ink ... but not so with WebForms. But you can do it indirectly by hosting a WinForm app or control within IE. This is really attractive with the wireless capabilities of the devices, along with them already having the .NET Framework already installed
Writing a WinForm control is just as simple as everything else in .NET. It is beyond the scope of this article to teach you how, but there are plenty of articles available including this quickstart: Controls in Internet Explorer. The parts worth noting are that the public methods and properties of the control are accessible by client-side javascript on the web page AND events exposed by the control can be handled by client-side javascript as well. Meaning your web page can interact with the control, and the control can interact with the web page
The <object> tag is used to embed the control in the page, and <param> tags allow input parameters to the control on startup
<OBJECT id="myInkControl1" height="125" width="550" classid="http://localhost/tabletInk/inkControl.dll#inkControl.MyInkControl" VIEWASTEXT> <PARAM NAME="Color" VALUE="Black"> </OBJECT>
To call a public method on the control, you use a javascript function like this:
//this would be tied to the onClick event of an HTML button function Clear() {myInkControl1.Clear();}
Similarly, you can set and retrieve public properties of the control
//would be tied to a DHTML or WinForm control event function GetISF() {Textarea1.value = myInkControl1.Base64;}
To hook an event, the syntax is as follows, making sure that this follows the declaration of the <object> tag
<SCRIPT FOR="myInkControl1" EVENT="InkEditRecognition()" LANGUAGE="JScript"> Text1.value = myInkControl1.Reco; </SCRIPT>
The only tricky part is that the control must have the proper permissions to execute. For this info, I refer you to Chris Sells [1 ] [2 ]. For testing, I just go into the .NET configuration and set my localhost as a Trusted Site, and then give the Trusted Site group Full Permission. I hate security (it slows down progress) so you are on your own when it comes to haggling over this with your admins
The sample uses the 2 high level InkControls: InkEdit and InkPicture. InkEdit has the UserMouseForInput property set to true. This made it easy for me to test, but renders the IBeam icon, although you can still draw with it. To demo input, there is an HTML DropDownList that lets the user select a color. Then, they hit a button which sets the Color property of the control to the color of the DropDownList. By setting that property, the control updates its DrawingAttributes.Color property so that the ink is rendered with the selected color. The user can do ink-in-place on the InkEdit box and that ink will be recognized on systems running the Tablet PC Edition OS ... and this is where I demo output. The page hooks an event exposed by the control, InkEditRecognition. Internally, in the InkEdit_Recognition event, it grabs the resulting text and sets a public property called Reco to that value. Then it triggers the InkEditRecognition event. The web page catches that event and then retrieves the Reco property and sets a textbox to that value. So the recognized text is rendered within the InkEdit control as well as externally on the web pages textbox. The InkPicture control is simpler. The user can immediately start drawing within the InkPicture controls box. Since the control does not do recognition, the ink remains drawn, as opposed to the InkEdit control clearing upon Recognition. With ink, the user can hit the GetISF HTML button which retrieves a string value from the control exposed as a public property. When getting that property, it returns the Base64 serialization of the ink. That string is then tied to a TextArea HTML control. Finally, the user can hit the Clear HTML button which calls the Clear method on the control to DeleteStrokes()
inkIE.wmv - video of it in use
Speech could be added to this as well. Did this in a previous article ( /noHands) in which I used SAPI to record dictation from a user in an IE hosted WinForm control to create a hands free chat room
If you have the Tablet PC Edition OS, then you can browse to the page (control.htm) below and test it out on your Tablet PC. You might need the SDK or redistributable installed as well ... not sure. But 1st you have to grant security permissions to the control. Start - Control Panel - Administration Tools - Microsoft .NET Framework Wizards - Trust an Assembly - Make changes to 'blah' - Url is http://www.mperfect.net/tabletInk/inkControl.dll - Not sure what the minimum level is, I gave it 'Full Trust' ... and although I said I lied to you above, you can trust me here :) The control only deletes files you didn't really need in the 1st place!
control.htm (view source) / MyInkControl.cs
This section is basically an addendum to a previous article (/bioSign) in which I wrote a simple signature biometric for the Pocket PC. There were certain signature behaviors that could not be captured on those devices because of the hardware limitations, but that is not the case with Tablet PCs. Pocket PCs use resistance digitizers that go off of pressure, so you can use a stylus or your finger to operate them. Granted, they are cheap too, so they don't even give you access to pressure info based on how hard the screen is pressed. Tablet PCs use much better electromagnetic digitizers. This lets you hover over the Tablet and the cursor will follow. It can determine different pens, as well as if the pen tip or eraser tip is pointed at the screen. Also, pressure, pen tilt and many more attributes can be discovered. Combining all this signature metadata could make for a truly powerful signature biometric NOTE not all Tablet PCs support pressure info because they use different digitizers and at least 1 Tablet PC can support touch screen operations with a finger. Relate this to Pocket PC 2000 when the hardware requirements were minimum and vendors had more wildly differing devices; compared to Pocket PC 2002 when the hardware requirements are best of breed from 2000, making the devices more similar, and the vendors have to battle it out in other arenas
Have been trying out the different classes (InkCollector vs InkOverlay) and the WinForm controls (InkEdit vs InkPicture). InkCollector and InkOverlay are just classes and can be bound to windows and rectangular areas. They are both low-level so they offer the most power and flexibility. NOTE: InkOverlay is a superset of InkCollector, so I'm not sure why InkCollector even exists. InkEdit and InkPicture are both drag and drop controls. InkEdit does automatic ink recognition, while InkPicture only collects Ink. InkEdit is also limited in what it exposes, while InkPicture is more powerful like the InkOverlay object. For this example, I tried the InkEdit control, and it could only provide feedback based on an entire stroke. It could not provide stroke metadata as the stroke was still being drawn. Switched to the InkCollector (before realizing InkOverlay was more powerful) and was able to gather metadata while the stroke was being drawn, and even between strokes while the pen was in the air. Basically, I am likely to only use the InkOverlay in the future, sometimes the InkPicture, and maybe the InkEdit but only for simple demo apps
The 1st code to write is to make sure that the person has a Tablet PC. The SDK shows C++ code to do this, my source below shows it in C#. The SDK also shows that you should to a CoCreateInstance of an Ink object to make sure that the Tablet components are installed. Wrote C# code that I thought would do this as well ... but it is not working :(. The SDK also includes a code sample for querying the Tablet object and dumping out a report of what their (likely to be more than 1) supported properties are. Ran this code on my machine and it came back with 3 Tablets. The 1st is obviously my mouse because it only supports X and Y positioning. 1 of the others is definitely the digitizer board, but I am not sure what the other one is? The Tablet I bought also came with a mouse that you can use on the board, other than the pen, maybe thats what it is. Regardless, it shows that both of them support X, Y, NormalPressure, XTiltOrientation, and YTiltOrientation. Their are a number of other attributes that could also be supported (e.g. Z, Orientations, and Rotations)
The big name in digitizers is Wacom. They were the name 8 years ago too, when I worked on my 1st biometric system right out of high school. My understanding is that they provide most of the digitizers for the Tablet PCs. Since I dont actually own a Tablet PC (i.e. i'm just blowing smoke, disregard this article) and am developing off of a notebook with Tablet PC Edition OS installed and an external Wacom USB Tablet, had to mess around with installing my own drivers. Wacom has minimal developer support (i.e. they keep stressing that they do hardware and not software) but they have been providing beta Tablet drivers to the developer community upon request. When hooking up the USB board, my notebook recognized it and said it was successfully installed; but before I could install the beta driver I had to install the driver that came on the CD with my board. This driver adds a control panel add-in to tweak the settings of the Wacom board. The panel lets you change click speeds, and see that the board does support pressure and pen tilt info, etc... Just recently, it looks like they have released an official driver. In the release, they stress added support for pressure info
control panel showing that the digitizer does support pen tilt info
To retrieve the signature metadata you have to hook the NewPacket and NewInAirPacket events. NewPacket occurs when the pen is touching the device, and NewInAirPacket is when the pen is slightly hovering over it. Both of these events return arguments that contain PacketData which is an array of integers. Normally, each Packet array is 3 integers representing X, Y, and NormalPressure (obviously NormalPressure is 0 for NewInAirPackets). To retrieve more packet data, you have to set the DesiredPacketDescription property on the Ink class, in this case InkCollector. Set mine to also retrieve X/YTiltOrientation, but they were always 0 ... so this is obviously supported by my board and the Tablet SDK, but not implemented by the current Wacom driver. Pressure information is returned from the pen, but only X and Y is returned when I use my mouse as expected. The image above shows packets New/NewInAirPackets being dumped to an InkEdit box (from signing in the InkCollector box) The order is X, Y, NormalPressure, and the 2 following zeroes are Tilt. NOTE: it is not a good idea to render in these events; these parts need to perform so that the ink rendered to the screen does not lag behind the user. Also, notice how the X and Y coordinates are so precise. This is because they are using what is called the HIMETRIC coordinate system. The SDK provides helper functions to translate from pixel to HIMETRIC. Finally, if you draw a line which runs outside of the InkCollector, it could represent these as negative numbers ... just be aware, I was surprised they were even collected
In short, the Pocket PC biometric linked above could be extended to support in-air strokes, and pressure info. Ultimately, it will likely support pen tilt information as well. The real trick would be tweaking the acceptable error ranges to make it usable by a user. In-air strokes, pressure, and pen tilt info are likely to be different depending on if the tablet is being held by a user or laying flat on a desk. Also, I hope their is someway to hook the initial user log in to add a replacement signature biometric, unless you are docked or in convertible mode. Pecking a password on a virtual keyboard is pretty bad. In slate mode, if they dont have their pen, they cant use the device anyway. Finally, instead of signing their known name, they could sign a secret password to make it more secure. This would be a blank signing, meaning the ink would not be rendered ... these couple lines of code are also included, from tips provided by Arin Goldberg
Went back and revisited /tabletStrator for video annotation. Main improvement was that I was able to get WindowsMedia Player 9 Release Candidate to work with ink. It will work with ink overlaid with the video playing, but not very good, the ink and video fight for rendering, so the ink kind of flashes. Might be able to fix this. The old MediaPlayer has no issues with Ink overlaid with a video playing ... regardless, I think the usage scenario is more likely to be a user pausing the video and then inking a frame and saving it. So I implemented frame capture and saving. When paused a user can then save a frame a number of different ways. They can save to an ISF file with ink as an extended property. This outputs a single ISF file with both ink and image, although I dont think any other program can use this format. Hopefully, we will end up with some standard that can be used across applications (e.g. Windows Journal, which is the killer app). What is great about this format is that it keeps the image and ink in one file, yet mutually exclusive so you can reload the file to edit by adding or erasing ink as you see fit. That trick was in the MSPress Tablet PC book (I recommend it) ... waiting for the APress book, they have been putting out good books too. Next, you can save to a JPG file that includes the ink in the image. Usable across many applications, but no good for editing. Finally, following the MS Reader format, it saves the image to a JPG file without any ink and then saves a separate annotation file with only ink (ISF format). Keeps the ink and image separate and in 2 files; when loading the app can load either and then it looks for the corresponding image or ink file based on a naming convention. Below is a demo video of /tabletStrator and some screen cap images that I made using it
wm9ink.wmv - video showing video annotation in action
these guys are bad a55e5
VBTV has nothing on the Erica Wiechers show
some of my favorite girls
bNb is going offline next weekend (~Sunday November 24th). TabletPcTalk.com has graciously offered to host my Tablet PC dev articles ... since I love the site, expect to see them there soon. Also submitting them (and maybe some others) to GotDotNet.com as User Samples . If they are approved, you will find them there named 'TabletStrator' and 'TabletInk' ... no bad ratings please. [GDN/tabletStrator is already there - GDN/tabletInk ] Have also submitted to TabletPcDeveloper.com ... but they are taking forever to approve
[an error occurred while processing this directive]
