Leap Motion Depth Sensing Camera for Physical Therapy #leapmotion

Over the last couple of months I have been working with a development version of the Leap Motion camera, in collaboration with SilverFish Studios, to build VR prototypes for physical therapy focused on finger and hand exercises. We set out to create simulations of actual clinical exercises that we could demonstrate to physiotherapists and occupational therapists to obtain feedback on the potential use of the Leap camera in the home. Three prototypes were constructed and are shown in the video below. The virtual tasks comprised: Cotton Balls (lift virtual objects and place them in bowl), Stacking Blocks (stacking virtual blocks), and the Nine Hole Peg Test (lifting and moving of nine virtual pegs).

We decided to use a fairly high level (easy to use) game development tool that facilitated quick prototyping, had a research/indie friendly licence, and had plugins for both Leap and Kinect. Shiva3D was the engine chosen and we were able to create both Kinect and Leap interactive prototypes very quickly – its a shame that the Stonetrip, the company behind Shiva3D, are now in liquidation but Unity3D is another excellent engine for this type of work (despite the limitation of requiring the pro license for plugins). A good game engine provides many useful tools for building effective 3D interactions such as built in lighting and physics. The simulation of Jenga that we built provides a very good illustration of what can be constructed in a few days:

We were very happy with the trials and the response from clinicians at the Brain Injury Unit in Musgrave Hopspital, Belfast, and we hope to present the details and results of the trials at the ITAG conference this year. Physiotherapists and occupational therapists at Musgrave already use game based therapies quite a lot and were very positive about the potential use of Leap based virtual therapies for home use and particularly for younger patients. Clinicians provided a lot of very specific feedback on particular finger/hand/arm motion that they required including exercises that encourage patients to extend out their arm and open out their hand at the same time – like they were pushing someone away. We have already started on a gameplay mechanic that can include this motion and the video below shows a very early version:

The grab gesture is also very easy to implement with Leap and the following video demonstrates a simple shelf stacking task that could be the basis for a rehab game:

SilverFish Studios Release Fly Casting App. for iPhone

SilverFish Studios is an independent software company based within Northern Ireland. SilverFish will have been operating for 2 years come June – no mean feat in the current financial climate. They have had many ups and downs but SilverFish provides a fun working environment and the team are highly motivated, talented and supportive of each other. They have been involved in designing and making several games (one due for release soon), been teaching game development for local schools, and have been engaged in quite a lot of web development and consultancy – typically they enjoy taking on jobs that require aspects of problem solving and innovation.

They work hard and no more so in building Fly Cast Master which is an iOS app designed to help (beginner and experienced) anglers improve their technique by analysing their casting motion (while holding the phone). Fly Cast Master was a complex App. to make due to the design and construction of robust motion detection algorithms and the necessary collaboration with experts from the angling domain for data collection, testing, and in the provision of suitable training materials – trainers, equipment suppliers, anglers, and collaborators at Irish Angler magazine.

My wife, Therese Charles (or @bionic_rez), established SilverFish with the support of her father Brian McGinnis who was her greatest fan. He would be incredibly proud of what she has achieved with the company and particularly in achieving the release of Fly Cast Master. I’m involved with SilverFish in an advisory role – mainly over a working lunch or after hours – but it has been very educational and enjoyable for me to be even superficially involved in helping a software product reach the App. store. I’ve learned about the process and this will help me in my future teaching, research and consultancy.

With the support of the local and international angling community and industry there are a number of ways that Fly Cast Master can be taken forward, and SilverFish have a number of plans in place already. So I hope that you will download and have a go at Fly Cast Master – see if you can beat the top score! One of the great things about Fly Cast Master is that you can still practise your casting even if you can’t get out! If you are enjoying the App. or have any constructive feedback let SilverFish know – they are passionate about making the angling trainer as educational and fun as possible.

IEA/AIE Paper Acceptance

We have had a paper accepted to The 26th International Conference On Industrial, Engineering & Other Applications Of Applied Intelligent Systems (IEA/AIE) June 17-21, 2013 Amsterdam (The Netherlands). The abstract is copied below – this is only a part of the unpublished work that we have on the back of Richard’s PhD, and hopefully we will find time to work on a journal paper that wraps this work up nicely. David will be attending and presenting the paper.

TITLE: Facilitating Player Interaction in a Dynamic Storytelling Environment

AUTHORS: Richard Paul, Darryl Charles, Michael McNeill and David McSherry

Abstract.  Facilitating player interaction with stories generated using artificial intelligence planning techniques is an important challenge to be addressed in the development of interactive computer game worlds. The problem that we focus on within this paper is the loss of story context that occurs when plan steps are reduced to primitive actions which can be executed in the game world, making it difficult for players to understand their purpose. We propose a solution to this problem based on a mechanism for dual representation of story plans at the levels of abstraction required for meaningful player interaction and to enable plan steps to be executed in the game world.


Recent journal publications

Recently we have had three journal papers accepted for publication on the basis of collaborative research that has gone on over a several years. I think this sort of research is crucial, and very worthwhile, though the government’s REF research quality measurements do not really encourage this type of work (due to measurements being explicitly within discipline based units of assessment).

Collaborative journal papers are often years in the making – time to establish a group of researchers,  months to conduct the research, more months to write the paper, and generally many months to have the paper reviewed, amended and then hopefully obtain final acceptance. There is then some time before the paper actual appears as a publication.

If you are not familiar with the process you will understand why we were delighted to have the following three journal papers accepted on the back of years of work:

PhD games research topics on offer at Ulster

Two of our games related PhD project proposals have gone live on the University of Ulster website:

Deriving Context-awareness in Quality of Service for Real-Time IP Cloud based Multiplayer Gaming


Motion and Camera Sensor Data Fusion for Personalised, Adaptive and Interactive Rehabilitation Connected-Health Systems

The 1st topic continues research by Stephen Workman and connects with state-of-the-art work currently being undertaken by Prof Gerry Parr and his group. In this PhD the student will investigate mechanisms for the enhancement of cloud-oriented streaming game services.

The 2nd topic moves on from successful PhD work of James Burke and other VR and games rehab research at Ulster. The particular focus of this PhD will be on the fusion of sensor information from motion and camera sensors to develop more personalized and adaptive physical rehabilitation interactive systems.

In the past we have offered machine-learning in games, intelligent interactive story-telling (MMO games context), and games enhanced learning PhDs that have been very successful. I would be interested in discussing future extensions to the work to suitably qualified and motivated candidates in the future.

Funding is available for suitably qualified candidates either from within Northern Ireland further afield (difference funding sources). Shortlisted applications are interviewed and after this process selected candidates are offered funding (conditional on meeting min. educational requirements).

Contact me at dk.charles@ulster.ac.uk for further information

GDmag Frontline Awards

The Game Developer Frontline awards for 2012 have been announced. The winners were:

  1. Art tools: modo 601 by Luxology
  2. Audio: Pro Tools 10 by Avid Technology
  3. Free: Blender (open source)
  4. Game Engine: Unreal Engine 3 by Epic Games
  5. Middleware: Havok
  6. Programming: Mozilla Foundation

In addition to this, Unity3D received a Hall of Fame award, perhaps setting it aside from competing game engines Unreal Engine, CryEngine, and others. I have been using Shiva3D, which hasn’t been nominated, though has favourable license/cost terms for small companies/indies.

modo 601 was an interesting winner in the “art tools category”, beating established products such as 3ds Max, Maya, and Photoshop. It seems to have integrated a range of tools for 3D art asset production that sets it apart – I hope to have a play with this one day. I really don’t know enough about audio to comment on this section, but Blender winning an award in the “free” category says a lot for how far this tool has progressed. My wife’s company, SilverFish Studios use both Blender and Gimp commercially, and they find them very productive and flexible tools. In my teaching I use this these programs for graphics production as well as nominee Box2D physics within a range of game development tools. Ogre3D was also nominated in the “free” category, we have also used this tool in our teaching and research – it great to see Ogre3D still doing so well.

GameMaker was nominated in the “game engine” category, reflecting the fact that it has made a significant impact this year with its new capability to create games on multiple platforms. Personally I would say that the new GameMaker :Studio has impressed me most this year. However, the development software that I have used most over 2012 was Photoshop and Visual Studio. Both excellent tools.