Gesture technology, invisible user interface and the evolution of human-to-machine interaction

The future of human-to-machine interaction will be a move away from touch screen technology to systems which anticipate natural movements and can be controlled by gestures, voice and ambient communication, according to global design and innovation consultancy Fjord.

Originating in 2001 with a focus on digital media, the company aimed to go beyond using digital for marketing communications, to focusing on services that people want and need in their lives.

In 2013, the company was acquired by Accenture and has since grown, so now Fjord within Accenture Interactive has expanded to 18 locations worldwide with a team of 700 people.

Today, Fjord works on systems which change the way people interact with technology, digital media and the world around them.

Marketing sat with Olof Schybergson, Fjord founder, and Bronwyn van der Merwe, Fjord Australia lead, to discuss the history of the company worldwide and in Australia, as well as the technology and the opportunities it presents.

Marketing: Please explain gesture technology and some of the other technology you’re working on.

Olof SchybergsonOlof Schybergson: What’s interesting about gesture technology actually goes beyond that specific technology, to a set of different ways that we interact with technology. Go back to human computer interface, engineering and design. For the longest period of time the way that people in general interacted with technology was essentially indirect interaction. Using a mouse to control a cursor on the screen and direct it that way. The smartphone really brought about a new wave of interacting with technology which is the touchscreen, that we now are all really accustomed to. Our kids, old people, anyone in between, are used to using touchscreens. That is now the standard, the dominant way that we interact with technology.

Alongside that dominant interaction paradigm, the shift from the PC or computer to the smartphone as the digital hub in your life has been very strong. To move from a bigger device, to something that you carry in your pocket. Those two things have changed how we interact with computing.

Now we see the evolution of gesture based technology, voice recognition, artificial intelligence, ambient interactions and computing and signalling. A whole range of new ways of input and output between technology and people that is actually starting to add to and make the interaction that we have with services, with brands, with technology, potentially more natural for people, but also breaks that very strong dominance of the touchscreen as the only real way of interacting with technology.

In very normal interactions day-to-day, we’re already using gesture based technology, and then we’re also starting to see it integrated more into digital experiences, one kind or another. We’re also very excited about voice technology, voice recognition and voice interaction. That’s a very human way of interacting. Our voice is a big part of our personality, how we communicate, how we talk to people, and increasingly the technology is very natural. It’s much more natural actually than touch screens or other form of typical interaction with technology. We see the emergence of our voice in lots of different places.

M: What are some of the potential setbacks that the new technology has been facing in terms of integrating into mainstream use so far?

OS: As you bridge towards more natural forms of interaction, for example, voice recognition and technology talking to you. What’s interesting about that is that as humans, we start to apply the same expectations as we do with human communication. If the content is right, but it’s said at the wrong time, it feels a bit too intrusive too quickly. If it’s using the wrong tone of voice, all of these things are small nuances that we pick up on in our human-to-human interaction very quickly and very easily. And now we start to use the same filters with technology. It means that it’s really challenging actually, to get all of those things right, at all times, with technology. That’s one of the inherent challenges within it.

bronwyn1Bronwyn van der Merwe: An interesting challenge with gesture, is that there are cultural significances for given gestures. There isn’t necessarily a universal gesture that works with every culture. Part of what we’re doing at Fjord is exploring that space and thinking about what other gestures that we can use, and where do they need to be different for different cultural situations to control the world around us.

M: Gesture technology and invisible user interface will change the way people interact with technology, can you think of any ways that it may change ways people interact with each other?

BvdM: One of the key things that invisible UI or natural interfaces change our lives is that they stop the dominance of the screen in front of your face. In a family environment, for example, with Amazon Echo, which is a voice operated device that allows you to turn music on and off, ask what the weather is, find out all sorts of information. Within a family environment, that immediately changes the dynamic, because you no longer have to stop, look at a screen, you can interact with your children and you can make voice request to Echo and it’ll give you that answer. It changes the dynamic in an environment where the screen is less dominant.

OS: That’s a really important point. The social dynamic with a screen tends to be personal. You cocoon into your own space, and that might actually pull people apart rather than pushing them together. We have a colleague at Accenture who bricked Amazon Echo, this little device, into his home, and told his kids, ‘let’s use that for getting information from the internet, let’s not use our screens at home’, and it completely changed the dynamic in the household for the better.

As digital is becoming more integral in our lives across everything, whether it’s shopping, healthcare, exercise, insurance, banking, finances, across everything that we do, digital is becoming really central. As that happens, people are also recognising that they crave for deep human interaction and experiences. You have this counter trend that we call digital dieting which is finding the moments when you disconnect from screens, finding the moments where you put technology to the side or use technology only to enhance the face-to-face interaction rather than allow it to disrupt.

M: How are early examples of these technologies performing in the transition from research and development to mainstream use?

OS: When you think of these technologies it’s really, really promising and very interesting technologies and still in the R&D labs. There are lots of technologies that are out there, in prevalent use. If you think about voice, for example, all the digital native leaders on the west coast of the US: Microsoft, Facebook, Apple, Google, and Amazon, all of them have their own version of artificial intelligence through personal assistant and voice is the typical way – not the only way, but the typical way – to interact with them. That’s now used in a combined way by tens or hundreds of millions of customers, just from those five providers alone. So it is one way or another definitely filtering out into the mainstream and not only in the R&D labs.

We recently did an engagement with Philips using very advanced technology where essentially you have a brainwave reader, that allows ALS sufferers that have an advanced form of ALS to have a slightly more dignified life, and have a little more control over their lives, than they would be able to have otherwise. It allows you to do simple things like turn on and off the lights in a room, send pre-programmed messages, turn off the TV, move your wheelchair etc. Not by talking, not by using gestures, but through thinking. So it picks up on the patterns in your brain, and you can train your brain to focus on certain things, and that allows you to perform those type of actions. A person that has lost complete ability to use their body, including their voice, that could be a truly transformative technology.

M: What opportunities does technology like this present for companies looking to engage with consumers?

BvdM: If you think about the proliferation of touch points today, in terms of all the physical and digital interactions that people have with the brand, as we move to a place with more and more touch points it becomes more and more complex. So brands need to think about how to design really beautiful, natural and effortless interactions with their consumers.

Thinking about natural UIs (user interfaces) gives brands a ways to connect with consumers in a way that they previously haven’t been able to. In the 1990s we had the rise of the internet, in the 2000s when Fjord started we had the rise of mobility. What we see now is a move towards living services, which is a set of services which are much more highly contextual and highly personalised to individuals. They’re picking up on all of the information that is being collected from all of these different senses in the environment, through voice, through gesture, through touch, through digital interactions, and they’re learning about you. As they learn about you they’re becoming more and more personalised, and start to adapt and flex and become really intelligent. We see that this is going to be a whole new breed of services that really change the world in a profound way and they change the way people will interact with different brands.

We’re doing a lot of work with different companies to help them to understand and meet the challenge of living services and how they need to evolve to stay ahead of the game.

OS: To build on that, I think that the change for marketing communications professionals, their career and their focus, can actually be quite profound. Advertising and marketing has gotten to a stage where the same message for everybody is not necessarily working as well as it has been in the past. People are saturated with marketing messaging and they feel overwhelmed. The opportunity for the marketing and communications professional is actually to start to blur the boundaries between marketing and service delivery.

As the boundaries between delivering a service and marketing a service are eroding, good communication needs to be a part of all aspects, whether you’re selling something or you’re delivering a good service, and good marketing communication needs to be a constant thread throughout the experience for customer. Whether they’re evaluating and thinking about the purchase, or whether they’re using a service or contacting a brand or a company. That’s the opportunity, a shift into a product or service experience, being the marketing.

Therefore marketing communication professionals should demand from the company that they’re representing a great service, and product experience, because without that, you can’t do good marketing anymore. There’s a very, very strong symbiosis now. You can’t sell lies, you have to tell the truth, and if you tell the truth about a good product or service, then your customers will actually do a lot of the marketing for you.

M: Are there any other new technologies that you’ve been working on, or new trends that you can see arising in the very near future?

OS: A general trend that everyone is struggling with – very closely related to what we’ve just been talking about – is the fact that there’s more and more, all the time. More touch points, more communication, blurring of boundaries and so on. It’s a very complex space.

It used to be that you control your communication, you control your brand. Now, your customers control a lot of the communication and perception around your brand. Also, you don’t necessarily control the channels any more. It’s a mash-up of services and marketing messages in different places. Regardless of what perspective you look at, there’s a growing complexity. Navigating that complexity, both for the end customer, and end user, people, but also for professionals in this space is really challenging.

The opportunity there is to think of elegance and simplicity, not just as something that you apply to your communication, but as something that’s very strategic and important for you and for your customers. Those who consistently apply elegance and simplicity in their business, in their communications, in their interactions, are the ones that will win, in this world that is increasingly complex.

BvdM: Building on that, I think that what we have seen increasingly is the way to create elegant and simple services is through the process of service design. We’re seeing that businesses here in Australia and globally are turning to design and design thinking as a way to help them really solve some of their big strategic challenges that they’re facing. As disruption hits a lot of different industries, technology changes the playing field on a regular basis.

Customer expectations are really rising, so we’re seeing liquid expectations amongst customers, where it’s no longer enough for a brand to think about the very best customer experience that’s being provided by their competitors. People are now taking the very best of breed experiences that they have from all of the brands that they interact with, whether it’s a utility or a bank or an airline, and they’re expecting this brand to have that type of experience, that best-of-breed experience. We’re definitely seeing a rise in design thinking as a way of solving strategic challenges.

OS: To do something fundamentally transformative and successful, you do need to ensure that you cover three perspectives, there’s the technology perspective, there’s the people perspective, which is where design comes in, and there’s the business perspective. There’s the three. It used to be that you could get the competitive advantage when you come up with a better business model, it also used to be that you can get a competitive advantage when you have better technology than anybody else.

Those aren’t necessarily sustainable competitive advantages any more, so people turn to design to find that competitive differentiation. Understand who your customer is, what are their needs and behaviour really, what it should be in the future and then create the service, the solution, marketing communication, that matches that, and then you might get to a sustainable advantage over your competitors. That’s one of the reasons that design is rising to the top in terms of priorities for executives.

Ben Ice
BY Ben Ice ON 19 November 2015
  • Eshan Sarpotdar

    Very useful article about gesture technology as a part of user interface. You have clearly mentioned the different types of technologies involved for gestures as a part of the UI. However, Following up on this, I came across and registered for a webinar on “UI/UX best practices in CMS based web design” and gain insights on various practices, processes and design strategies to create and deliver a rich and exceptional UI/UX for your CMS based website, here is the registration link-

  • ole christian hanshus

    TEJERINA. Big lesson to you. Cut the crapt