Get input from local devices (other than mouse and kbd)

"What other devices?" I can hear you wondering?

Well, system sound input could be handy, for one. But I have something rather specialized in mind. As I've mentioned elsewhere (and see link in my sig), I'm trying to implement a biofeedback-based game in SL. I have a biometric device that collects skin conductivity level and heartrate data via USB on Macs and PCs. I'd like to be able to pass that information to the SL client and hence use it in-world. There are open-source programs which can capture the data, but no SL client-side API to hand the data to.

I think I could do this by transmitting the data via TCP/IP to a server (already have the code to do it), then importing through XML-RPC, but based on what people in this forum have to say about XML-RPC, I'm not very confident that it will work or be responsive enough for what I'm trying to do.

If I could get something like this working, I've got a friend who wants to be able to bring his brainwave device data in and use it the same way.

What I'm wondering is, are there enough other people who want to be able to bring in *some* kind of locally-generated data to make an API of this kind worth requesting? What would other people use it for? Streaming audio is supported, so being able to hook your mic directly into the client seems doable. MIDI input, perhaps? Video seems kind of high-bandwidth, though it could be interesting to have slow framerate webcams into SL. I have the impression things like flight simulator joysticks and steering wheels are usually interpreted by the OS/game as keyboard or mouse maps, but if not, those are possible controllers that could be added to SL with this. I've seen a USB-based "sword" controller for one of the console boxes, possibly also for PC, that the medieval gamers and Star Wars fans might like.

Actually, even being able to capture keyboard input directly to scripts could be a boon in itself-- currently one has to use chat channels, am I right?

Could the API for this be made generic enough but powerful enough to be worthwhile? Is there other interest?

Comments welcome,

neko

I would pull my fingernails out one by one in exchange for good analog joystick support, especially dual-joystick game-controller type input. (one stick for analog movement, one for camera control)

Yes, that would be a big improvement. And if we had this kind of API interface, someone would probably write that package and distribute it.

neko

Unfortumately, it'll take a lot of alot of "enough." According to Robin Linden, an API isn't a very high priority. I say that not to discourage you, but as a notification that those of us who want an API need to keep generating these ideas and excite other people.


Well, if webcams came with proper API's -- I'll save that rant for another day -- then you could use your webcam as something of an iToy (or is that eyeToy?), like the Playstation does. It think that would be fun. If you could get a camera to track your head, and turn your AV accordingly. Now that would make SL immersive.


Well, if you "drive" a car, then technically the script is capturing keyboard commands for things like forward, reverse, etc. I'm honestly not sure, though, if there's a limit to the number of keys that a "vehicle" script can use.

I've seen Robin's reply. It was discouraging, but I refuse to give up hope! Philip seems to be more supportive, though I'd like to see substantive steps in the right direction (e.g. fix XML-RPC).

Jarod, I think you have good ideas for how to use local data. We'd need an API and an SDK, probably, too.

I was going to add a poll to help back this up, but I guess it's too late for that now in this thread. I'm not expecting this in 1.6, obviously, but it would be nice to have a bit more commitment than we've seen so far.

neko

CONTROL_UP, CONTROL_DOWN, CONTROL_FWD, CONTROL_BACK, CONTROL_RIGHT, CONTROL_LEFT, CONTROL_ROT_RIGHT, and CONTROL_ROT_LEFT.

So, basicly, page up, page down, forward, backwards, left, right, and shift+left and shift+right.

Oh, and CONTROL_LBUTTON to capture left-clicks from the mouse, or something like that, but I don't like to take that one as I tend to make my vehicles touch-responsive to mouse clicks.

Let's not confuse two things, though. While an "API," as I read, was not a high priority - I DID read some time ago that configurable keybindings were of relative import to the Lindens. I think you can see what I'm getting there.

Perhaps it's time I shot that over to the Linden Hotline.

I fear that you'll only ever get non-helpful CR replies in this technical area. In effect, the wrong Linden people are responding to the questions.

Quite a large number of suggestions have dealt with UI issues. I suggest that all of these ideas get attached to one or more "XML the UI" threads, of which there are several, since it then becomes easy to make all keyboard and mouse mapping go through an XML-specified mapping of local UI controls to client actions.

I think that Jarod is right, it's just not going to happen, and the voting system now makes it even less likely.

I've just posted an item on "Voting system acts against technical suggestions" in the General forum, which relates in an obvious way to this issue.

LL don't have even a fraction of the manpower needed just to squish their bugs, let alone to work on the business end of the servers and advance the state of the client as well. Short of getting the client open-sourced so that the community can add all these great features, I really don't see any light at the end of this particular tunnel.

I'd like to broaden this discussion out a bit and focus on an SL implementation of a flexible control/data protocol that could really enable all sorts of interesting interfaces to SL.

Open Sound Control is a good example of one.
http://cnmat.berkeley.edu/OpenSoundControl/

"OpenSound Control ("OSC" is a protocol for communication among computers, sound synthesizers, and other multimedia devices that is optimized for modern networking technology"

It's basically a simple message format using either stream or packet based transport (TCP or UDP) that can be configured for whatever the device or application need.

Spec: http://cnmat.berkeley.edu/OpenSoundControl/OSC-spec.html

Application areas: http://cnmat.berkeley.edu/OpenSoundControl/application-areas.html

I'd be interested to hear your thoughts.

komuso

BUMP

Surely some of you developers/designers out there think this is just a touch more important than some of the more mundane items which are floating to the top of the feature voting list?

OSC looks good to me. It's a published standard, it can use UDP (which SL uses), and it could easily handle all sorts of interesting device data, e.g. steering wheels, foot pedals, VR gloves, joysticks, MIDI devices, console game swords, guns, and of course those biofeedback devices I'm always going on about. Just imagine, if we ever got the ability to animate finger movements, you could pull on a couple of VR gloves and have your av match your finger movements as you perform on the guitar and stream live into SL.

I'm not as cynical as Morgaine about the voting system. Vote for Prop: 203 - Support local devices beyond mouse and keyboard!

neko

Please take a look at my post here:
/54/83/52018/1.html

It's about controlling an avatar via an iSight.
We have the motion tracking down... and can "broadcast" keystroke information to the world. We just need to figure out the Second Life end of thinks.

I think the answer might lie in XMLRPC. Any ideas? Anyone have any advice about this technology relative to MAX/MSP/JITTER?

Thanks.

Audrey.

If you have an iSight to keystrock emulator in place then you don't need to do anything, just map motion gestures to keystrokes and use as normal.

The current XML-RPC implementation is too slow to work with responsive real-time control.

0.3cents

Bi-annual bump.

Here's a suggested API that LL could easily implement, and which would allow easy implementation of input plugins to Second Life.

llInputStream(key agent,integer port,string terminator,float timeout);
Agent is the avatar whose client is going to use the plugin.
Port is an integer in the range 0..9.
Terminator is a single character or the empty string.
Timeout is in seconds.

the SL client will open a localhost TCP port in the range 7990 through 7999 (or similar range), and read lines (terminated with the specified character) or complete XML segments (if the terminator is the empty string). Each complete line or segment is transmitted as a UDP packet in the user's normal input stream, just as if it were a keystroke. If the timeout is non-zero, partial input will be sent after the timeout expires.

the script has to have PERMISSION_OPEN_PORT which is automatic for attachments on the agent, or for llAvatarOnSitTarget();

The script will receive input(integer port, string text); for each message.

Using TCP input on localhost allows for maximum flexibility and portability, since all systems capable of running SL have similar local socket APIs, as well as existing compatibility libraries and scripting languages.

Raw ANSI escape sequences could be sent by setting the terminator to ESC and the timeout to 0.5s, so you could simply telnet to the port from CMD.EXE, Hyperterminal, Terminal.app or an Xterm and hit function keys. The LSL script would receive ANSI sequences with the leading ESC elided.

BUMP

I'm glad to see others asking for this. It's right at the top of my list of requested new features!

Spinach et al, if you've not already done so, please look at the prop (203) and vote.

We're only about 80 or so short of a Linden comment I believe, so its in everyone's best interest to do so.

*waves* :link:

Have a call llForwardPort(key avatar, integer port, integer channel, string terminator, integer timeout);. Anything said on the channel would be forwarded to the matching port on the avatar's computer, and anything that comes in on the port would be forwarded to that channel, with the same terminator and timeout parameters as in the previous suggestion.

At this point, I'd settle for faster XML-RPC (current latency is 2-4 seconds at best, nothing like what would be needed to add a more interesting controller). But the port through client idea would be excellent, if implemented.

I've just shifted most of my votes to Prop 203. We're sitting at 223 votes right now. Please help! There are so many great things we could do with this functionality.

neko

Welcome back Nekokami, though apparently at a bad time (I saw your Answers post and sympathize, enough so I've also dropped to Basic).

I think many at LL believe that the new HTML request format will do better than the existing XML-RPC, but from what I've heard that is also bugged rather badly too.

I did get to see Carlos' latest incarnation of the Mandala Bead Temple last Wed, it works rather well despite the limitations.

--Alan

Alan,

We've survived bad times in SL before, but it remains to be seen if we'll get through this one. As I posted in Answers, I've suspended my daughter's participation from Teen SL indefinitely, for obvious reasons. (And no, I'm not letting her onto the main grid instead.)

Back to the topic in this thread, Komuso looked at the HTTP functionality briefly, and said he didn't think it would solve our problem, but I'd love to hear otherwise from someone who knows more about passing data through HTTP. My impression is that this channel, too, is not designed for rapid transfer of many small packets of data, which is what we need.

neko

I think there's two discussions that need to go on here.

First, the one for a resident-implemented local control protocol on top of the existing client. Either by injecting keystrokes, or using HTTP or RPC to create a short-lived TCP connection to transfer one control.

Second, one for Linden-implemented extensions that allow applications on the local PC and scripts in attachments or vehicles to communicate over the user's session.

The first really belongs in the scripting forum, and include code if possible.

The second would be an actual feature suggestion.