Artículo de investigación científica y tecnológica
Revista Colombiana de Computación
Arturo Moquillaza1, Freddy Paz2
, Fiorella Falconi2, Raysa López3
1Universidad San Ignacio de Loyola, Lima, Perú.
2Pontificia Universidad Católica del Perú, Lima, Perú.
3BBVA Continental, Lima, Perú.
miguel.moquillaza@usil.pe, fpaz@pucp.pe, ffalconit@pucp.edu.pe, raysa.lopez@bbva.com
(Received: 29 June 2018; accepted: 05 July 2018)
El método de evaluación de comunicabilidad (CEM) es una técnica propuesta por la ingeniería semiótica que mide el grado de efectividad de la comunicación entre diseñadores y usuarios mediante la interfaz. El presente caso de estudio encontró que la literatura existente se enfoca en aplicaciones de escritorio o web. En este estudio presentamos los resultados de una evaluación de comunicabilidad en un Sistema de ATM. El caso se desarrolló con especialistas en HCI, quienes recibieron el requerimiento del BBVA Continental para definir cuál de dos propuestas sería mejor recibida por sus clientes, además de menos intrusiva para le experiencia actual de Retiro. La operación llamada “Retiro Seguro” consistía en ofrecer un micro seguro durante el flujo de Retiro. Se convocó a clientes reales y se les pidió que ejecuten tareas sobre ambas propuestas en un entorno realista. Concluimos que la segunda propuesta tuvo un mejor diseño, y propusimos una solución intermedia.
Palabras clave: Diseño centrado en el usuario (UCD), Ingeniería semiótica, Método de evaluación de comunicabilidad (CEM), Interacción Humano-Computador, Cajeros automáticos (ATM)
The Communicability Evaluation Method (CEM) is a technique proposed by Semiotic Engineering that measures the degree to which adequate communication is achieved between designers and users is achieved through the user interface. The case studies found in the literature focus on desktop applications or web applications. In this study, we present the results of a Communicability Evaluation on an ATM System in the Self-Service domain. The experimental case was executed by specialists in Human-Computer Interaction (HCI) at the request from BBVA Continental to define which of two user interface proposals would be better received by its customers, as well as being less intrusive to the current retirement experience. The operation was called "Retiro Seguro" and consisted in offering a micro-insurance during the navigation of a withdrawal. Current customers were asked to perform some tasks on both proposed interfaces in a realistic environment. We concluded that the second proposed interface was better at achieving design objectives and we proposed implementing an intermediate solution.
Keywords: User-Centered Design (UCD), Semiotic engineering, Communicability Evaluation Method (CEM), Human-Computer Interaction (HCI), Automatic Teller Machine (ATM)
According to the works by Moquillaza et al (Moquillaza et al., 2017; Moquillaza & Paz, 2017), BBVA Continental is an important Peruvian bank that is in the process of digital transformation, and is very interested in topics of UX, HCI and Usability. In that sense, they have launched a digital product called "Retiro Seguro", which is a micro-insurance that is offered to bank customers to purchase during a typical withdrawal for a nominal cost. In Spanish, the name is a play on words because it can mean either “Safe Withdrawal” or “Withdrawal Insurance”. The Bank, in its desire to improve its product, and to get more sales, had two improvement proposals to implement in “Retiro Seguro”, but did not want a negative impact on the withdrawal navigation experience. In this aspect, they requested from the UmetSoft research group for support to carry out some tests to validate the proposals. UmetSoft suggested using CEM, a method proposed in Semiotic Engineering to evaluate Communicability as an expected attribute in a user interface (Alarcón, Medina, & Villaroel, 2013). The experience was performed based on the study by Paz et al (Paz, Paz, & Pow-Sang, 2016) with real users in a laboratory with an ATM, in front of the application as such. . As a result, we were able to determine the degree of communicability of the proposals and the quality of the meta-communication between designers and users. In that sense, we recommended using an intermediate proposal based on the parameters of the Withdrawal as such. Finally, we provide this information and recommendations to BBVA Continental.
According to Paz et al (2016) (Paz et al., 2016), from the point of view of Semiotic Engineering, Human-Computer Interaction is perceived as a communication between users (receivers) and designers (senders).
According to the above, CEM is a method proposed by De Souza (Sieckenius de Souza & Faria Leitão, 2009), to measure the level of communicability of a software application. The intention is to determine the level at which users can understand the design of the interface, that is, the messages sent by the designers (Reis & Prates, 2012).
According to Moquillaza et al (Moquillaza et al., 2017), Semiotics Engineering focuses on viewing the Human-Computer Interaction as a type of communication, and as such, with all the elements involved; Designers are senders who communicate a message to users who are receivers through a channel that is the interface of the application. The messages are all the elements that the designers place in the interface for the users.
According to Paz et al (2016) (Paz et al., 2016), Communicability Evaluation Method (CEM) is a method proposed by De Souza, to measure the level of communicability of a software application.
This method allows designers to determine how well users understand the messages sent, identifying the communication interruptions that occur during the interaction.
In Semiotic Engineering, this method establishes that the HCI is a type of communication between humans, with the computer as intermediary. This interaction includes the participation of designers and users, and communication occurs at the moment when users utilize the system. If the purpose of each interface element is properly communicated, users will be able to achieve their objective by using the given system (Paz et al., 2016).
Using the CEM method, a test is executed with users; they are asked to interact with the software interface being tested. This interaction is guided by a set of previously defined tasks that users must execute during the evaluation. All the interaction must be recorded on video to capture both the screen and the user's face. Subsequently, the user's actions on the interface and facial expressions will be analyzed (Paz et al., 2016). The audio in recordings is also important for this analysis.
With this information as input, the method is applied in three phases: Tagging, Interpretation, and Semiotic Profile (Bim, Leitão, & Sieckenius de Souza, 2012).
In this phase, specialists analyze all the recordings made to observe the user's reactions and identify communicative failures. Evaluators should relate each evidence of breakdown to one of the 13 labels proposed by the method. These labels are expressed in natural language and in some cases are even said or paraphrased by the users themselves.
According to De Souza (Sieckenius de Souza & Faria Leitão, 2009), this labeling means: “Putting words in the user’s mouth”. These 13 labels are the following (Paz et al., 2016):
“I give up.”
“Looks fine to me.”
“Thanks, but no, thanks.”
“I can do otherwise.”
“Where is it?”
“What happened?”
“What now?”
“Where am I?”
“Oops!”
“I can’t do it this way.”
“What is this?”
“Help!”
“Why doesn’t it?”
In this phase, the specialists analyze everything identified in the previous phase and analyze all the problems of the interface. Each communicative breakdown is related to a specific category or class of HCI problem (Paz et al., 2016):
navigation
meaning assignment
task accomplishment
missing/declination of affordance.
In this last phase, the specialists must be able to identify the general meaning transmitted between designers and users. After this analysis, the specialists will try to recover the original communication objective of the designers, they will put themselves in the place of the designers, and they will answer in the first person the following questions (Paz et al., 2016):
Who do I think are the users of the software product I have designed?
What do the users want and need?
What are the users’ preferences according to what they want and need? Why?
How is the system I have designed for these specific users, and how they should use the system?
What is my design vision?
The participants were six real clients of BBVA Continental; some were providers and other employees of Bank. Their ages ranged between 22 and 60 years, all had previous experience using the ATM, but never before had they purchased a “Retiro Seguro”. To represent the participants, we used these IDs: P1 to P6.
The interfaces corresponded to two improvement proposals for the product “Retiro Seguro” (BBVA Continental, 2017a). This product consists of a micro-insurance that is offered for sale during the navigation of a withdrawal, as long as it is greater than S/100 soles (“sol” is the monetary unit in Peru). A typical withdrawal has the following flow (BBVA Continental, 2017b):
(1) Card and PIN entry
(2) Operation selection
(3) Account selection, currency and amount
(4) Confirmation screen
(5) Screen of successful operation
In the current “Retiro Seguro” navigation, as can be seen in (BBVA Continental, 2017a), a button was added in the withdrawal flow confirmation screen that, when pressed, activates the “Retiro Seguro” flow and returns the customer to the “Successful withdrawal” operation screen.
In this context, the Bank had two proposals: the first, mainly visual improvements without change of flow, and the second, with a change in the withdrawal flow, by replacing the safe withdrawal button with a full screen. With the second proposal, the Bank expected customers to buy more, but they feared that this new screen would have a negative impact on the withdrawal flow, the main operation performed in the ATM channel.
Next, we show the first (Figure 1) and last screen (Figure 2) of the withdrawal flow.
Figure 1. Current withdrawal flow: First Screen (screenshot)
Figure 2. Current Withdrawal flow: Last screen (Successful Operation) (screenshot)
Next, we show the flow of screens of Proposal 1 (Figures 3, 4, 5). These screens are those between Figures 1 and 2 and those that complete the Withdrawal flow.
Figure 3. Proposal 1: Button in Confirmation Screen (screenshot)
Figure 4. Proposal 1: Additional info after pressing the button 1/2 (screenshot)
Figure 5. Proposal 1: Additional info after pressing button 2/2 (screenshot)
Next, we show the flow of screens of proposal 2 (Figure 6). These screens are also between Figures 1 and 2, and complete the withdrawal flow. Note that the new “Confirmation Screen" would be Figure 7.
Figure 6. Proposal 2: Screen offering “Retiro Seguro” (screenshot)
Figure 7. Proposal 2: New “Confirmation Screen” (screenshot)
The Bank presented two navigation proposals, which were designed by its UX department. This department already formally or informally evaluates usability. In addition, we looked for a complementary approach that would allow us to evaluate the interfaces, having as the main objective not to hinder the cash withdrawal process from the ATM. For those reasons, we chose to apply CEM.
The communicability evaluation test consists of two activities: Perform both a typical withdrawal and a withdrawal by obtaining a “Retiro Seguro” product at the ATM with the first navigation and performing the same tasks on the ATM with the second navigation. We take into account the following for preparing the test:
The understanding of the screens that inform about the characteristics of the product.
Appropriate visibility of the “Retiro Seguro” option in proposal 1.
The impact of the full screen in front of a client that only wants to withdraw in the proposal 2.
In order to avoid some treds in the research, related to the proposals, we presented to each participant, the activities in different order.
Table 1 describes tasks in the evaluation.
Table 1. Description of the task in communicability evaluation
|
Activity |
Description |
|
Withdrawal in the first proposal. |
Performing a Withdrawal for an amount greater than S/100. Performing a Withdrawal greater than S/100 purchasing a “Retiro Seguro”. |
|
Withdrawal in the second proposal. |
Performing a Withdrawal for an amount greater than S/100. Performing a Withdrawal greater than S/100 purchasing a “Retiro Seguro”. |
The Communicability Test was performed on December 7, 2017. This study was implemented in the ATM laboratory of BBVA Continental. The interfaces were carried out in the ATM application and deployed in ATM Test environments to make the test experience as close to reality.
We used the following electronic devices and software:
Two ATMs NCR 6621 touch screen with the version of the screens in the local ATM application.
VNC remote control to access the ATM screens and record the interaction.
FastCapture Trial Edition to record the interaction itself.
Two mobile phone video cameras to record all the gestures, facial expressions and commentaries aloud of the users.
A VISA test environment card for user utilization.
The post-test applied to users contained the following questions:
Q1: Did you complete the tasks?
Q2: Do you consider that the information shown on the interfaces is enough and complete for buying the “Retiro Seguro”?
Q3: Do you consider that the information shown on the interfaces about “Retiro Seguro” is easy to understand?
Q4: Do you consider that the information shown on the interfaces about “Retiro Seguro” is useful and let you to buy a “Retiro Seguro”?
Q5: Do you consider that is easy to navigate on the ATM interfaces?
Q6: Have you felt oriented when you used the ATM for making the requested tasks?
Q7: How do you qualify your satisfaction degree in the use of the ATM for making the requested tasks?
Q8: Would you use the ATM again for the requested tasks?
Users had to answer in a 1-5 scale (from less to more). We applied the post-test at the end of the interaction of the user with both proposals. The Table 2 shows a summary of the results.
Table 2. Post-test results
|
Proposal 1 |
Proposal 2 |
|||||||||
|
Questions |
1 |
2 |
3 |
4 |
5 |
1 |
2 |
3 |
4 |
5 |
|
Q1 |
0 |
0 |
0 |
4 |
2 |
0 |
1 |
0 |
2 |
3 |
|
Q2 |
0 |
0 |
3 |
1 |
2 |
0 |
0 |
0 |
2 |
4 |
|
Q3 |
0 |
0 |
2 |
2 |
2 |
0 |
0 |
1 |
1 |
4 |
|
Q4 |
0 |
0 |
1 |
4 |
1 |
0 |
0 |
1 |
1 |
4 |
|
Q5 |
0 |
0 |
0 |
3 |
3 |
0 |
0 |
2 |
1 |
3 |
|
Q6 |
0 |
0 |
0 |
3 |
3 |
0 |
1 |
0 |
3 |
2 |
|
Q7 |
0 |
0 |
0 |
4 |
2 |
0 |
1 |
0 |
2 |
3 |
|
Q8 |
0 |
0 |
1 |
1 |
4 |
0 |
0 |
1 |
1 |
4 |
In Figure 8 we show the comparison of answers about both proposals. We do not identify significate differences between them.
Figure 8. Comparison of proposals
After studying the recorded videos and analyzing the interaction between users and ATMs, we identified the communicative breakdowns.
The results show that no events were registered for the following tags: “I give up”, “Looks fine to me”, “I can do otherwise”, “Where is it?”, “I can’t do it this way”, “Help!” and “Why doesn’t it?” They did not appear in activity 1 or in activity 2.
Tables 3 and 4 show the tags mapped in every activity.
Table 3. Tags on activity 1
|
Tag |
P1 |
P2 |
P3 |
P4 |
P5 |
P6 |
|
Thanks, but… no thanks. |
X |
|||||
|
And now what? |
X |
X |
X |
X |
||
|
Oops! |
X |
|||||
|
What is this? |
X |
X |
X |
X |
X |
Table 4. Tags on activity 2
|
Tag |
P1 |
P2 |
P3 |
P4 |
P5 |
P6 |
|
What happened? |
X |
|||||
|
And now what? |
X |
X |
X |
X |
||
|
Where am I? |
X |
|||||
|
Oops! |
X |
|||||
|
I cannot do it this way. |
X |
X |
X |
X |
The events for the remaining tags in the activity 1 are described as follows:
Thanks, but… no thanks.
P1 preferred not to take the “Retiro Seguro”, in spite of us suggesting to her that she to do it. She said she was afraid to press the button because she did not know about the product.
What now?
P2 and P5 did not initially understand what they had to do on the first screen of “Retiro Seguro”. In addition, P6 did not understand how to continue on the confirmation screen.
P1 was very confused with the sequence on the final screen; she was waiting for the money when she still had to make a final choice.
Oops!
There is not the option to withdraw S/10 soles bills, but P5 tried to withdraw that amount, so an error message appeared. She tried again by changing the amount, and then she went on.
What is this?
When P4 was asked to take the “Retiro Seguro”, she was not sure if it was the right button.
The situation of P1 and P3 was different because they were surprised to see the first “Retiro Seguro” screen, and they did not understand what it was.
Another was the case of P5 and P6, who did not understand what the confirmation screen was.
Now the events for the activity 2 are described as follows:
What is this?
At the beginning, P6 was confused and did not understand the first screen of "Retiro Seguro".
What now?
P2, P3, P5 y P6 were confused because when they saw the “Retiro Seguro” screen, they had expected the confirmation screen. Then, they did not know what they had to do next. P6 also had problems in identifying how to continue on the confirmation screen.
Where am I?
On an error screen, P1 made the gesture of pressing a button in a place where there was not any. She pressed the bottom right of the screen.
Oops!
P1 received an error message since the amount entered was not dispensable, but she corrected the amount and continued.
What is this?
P2, P3 and P6 were confused about what they were seeing at on the “Retiro Seguro” screen. Something similar happened for P5 when she saw the confirmation screen and had difficulty understanding what it was about.
According to the previous section, the main tags identified were “And now what??” and “What is this?” These two tags are related to two typical HCI problems: Navigation and Assignment of meaning. Both emissions are expected, given that these navigations are new, and users are expected to learn the new navigations.
In general, users finally indicated that they understood the interfaces and that they could complete the requested activities.
The actions that led to users making a mistake were not related to "Retiro Seguro". In both cases, the mistakes made were the entering of non-dispensable amounts. The information about limited currency denominations is shown to the user, but apparently, it should be clearer.
The users also expressed an interest in buying "Retiro Seguro", but only after knowing what it meant. The mere presence of the button was not enough to increase their interest. However, several users stated that a full screen could be invasive, and they would prefer not to see it as long as the amounts of their withdrawals were low since it would be an impediment to making quick withdrawals.
An additional issue not anticipated was the users' reactions to another existing functionality called "Retiro rápido". This problem has also been reported to the Bank, and a separate study on this subject is expected.
Finally, we wrote a report to the Bank containing the following findings and recommendations:
With proposal 1, users were afraid to press the "Retiro Seguro" button because they felt that there would be an additional charge for some type of service without first informing them about the process.
With proposal 1, users will rarely choose to purchase the "Retiro Seguro" product because users are not given details about it.
With proposal 2, users have received information about "Retiro Seguro" and there is a possibility that they will buy it more often if any of the following conditions occurs: withdrawal of a large amount of money, withdrawal in a dangerous place, or if the area near the ATM has a reputation for questionable activity.
According to the users, the information shown on the "Retiro Seguro" screen is understandable and describes the terms of the product well.
Three out of six users stated that they would not object to seeing the "Retiro Seguro" screen, but they will choose not to buy the product (they emphasized the usefulness of knowing about the possibility of insuring their money).
All six users expressed the usefulness of knowing the option "Retiro Seguro" for situations in which the purchase of this product would have merit.
With proposal 2, two of the users expressed a possible future annoyance due to the repetitive appearance of an additional screen, since they would not be interested, especially in cases where the withdrawal would be a low amount. At that time, however, they did not express any displeasure when the screen appeared for the first time. On the contrary, they highlighted the usefulness of knowing that the possibility exists.
Both proposals have the same degree of usability and communicability. According to the results obtained, it is not possible to establish a significant difference between one and the other.
R1: Identify those conditions where "Retiro Seguro" is useful for users, to specify and personalize the offer.
R2: Offer and explain about the service should appear at least one to all users, so that they know about the product and can opt to buy it when making a withdrawal.
R3: ATM waiting times could be used to display information screens about the product.
Once the recommendations were received, they were processed by the ATM development team to make them concrete actions. In this sense, in the final presentation, the Bank team added a slide in which they discuss the possible ways to implement the recommendations. They are the following:
For R1: This could be achieved by joining both proposals and configuring either an on-screen button or a full-screen explanation. We can start by evaluating the amount of the withdrawal, and then add more parameters such as ATM number, customer, schedule, etc.
For R2: This could be achieved by configuring the first version of the application the configuration of showing the full screen for almost all withdrawals amount options, and in a later update of the application, change the configuration to it should be. For example: first show the full screen for lower withdrawal amounts, such as S/100, but a few weeks later, change it to S/1,000 so the full screen only shows for larger withdrawal amounts.
For R3: This could be achieved by reviewing the material that is being shown to clients for “Retiro Seguro” at times of mass advertising, on the withdrawal receipt, and even in interactive advertising only for informative purposes.
This article presents the results of a case study in which the CEM was successfully applied to an option from the ATM application of BBVA Continental. The purpose of this study was both academic and strategically aimed at providing a solution to the Bank. It involved the participation of UmetSoft and the ATM Systems team of the Bank in mention. The objective of this analysis was to evaluate two proposals to improve the “Retiro Seguro” product and identify which ones had lower communicative breakdowns and better quality meta-communication. An analysis of the results allowed the specialists to conclude that the second proposal could create problems for the withdrawal of small amounts, so an intermediate solution based on the withdrawal amount was suggested. Likewise, it was concluded that the level of communicability of both proposals was acceptable.
Following the line of the conclusions of Paz et al in (Paz et al., 2016), where it indicates that more cases of study should be executed on applications in other contexts, it was found that this method works properly in a self-service context such as ATM applications. Likewise, we recommend that communicability evaluations be applied to more ATM options and to consider this attribute in general throughout the process of design, development and evaluation of interfaces for this application.
We want to thank BBVA Continental, especially the ATM Systems team, and UmetSoft for the support provided throughout the process until the completion of this work. We also want to thank Dave Kinskey for his constant support during the writing process of this paper.
Alarcón, C., Medina, F., & Villaroel, R. (2013). Finding usability and communicability problems for transactional web applications. In ChileCHI ’13 Proceedings of the 2013 Chilean Conference on Human - Computer Interaction (pp. 88–93). ACM. https://doi.org/https://doi.org/10.1145/2535597.2535617
BBVA Continental. (2017a). Retiro seguro. Retrieved December 14, 2017, from https://www.bbvacontinental.pe/personas/seguros/proteccion/retiro-seguro/
BBVA Continental. (2017b). Retiros: Paso a paso. Retrieved December 14, 2017, from https://www.bbvacontinental.pe/personas/canales/cajeros-automaticos/#ficha-content-2
Bim, S. A., Leitão, C. F., & Sieckenius de Souza, C. (2012). Can the teaching of HCI contribute for the learning of computer science the case of semiotic engineering methods. In IHC ’12 Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems (pp. 185–194). Brazilian Computer Society.
Moquillaza, A., Molina, E., Noguera, E., Enríquez, L., Muñoz, A., Paz, F., & Collazos, C. (2017). Developing an ATM Interface Using User-Centered Design Techniques. In International Conference of Design, User Experience, and Usability (pp. 690–701). Springer, Cham.
Moquillaza, A., & Paz, F. (2017). Applying a user-centered design methodology to develop usable interfaces for an Automated Teller Machine. In Interacción ’17 Proceedings of the XVIII International Conference on Human Computer Interaction. New York, NY, USA. https://doi.org/10.1145/3123818.3123833
Paz, F., Paz, F. A., & Pow-Sang, J. A. (2016). Application of the Communicability Evaluation Method to Evaluate the User Interface Design: A Case Study in Web Domain. In DUXU 2016: Design, User Experience, and Usability: Design Thinking and Methods (pp. 479–490). Springer, Cham. https://doi.org/https://doi.org/10.1007/978-3-319-40409-7_45
Reis, S., & Prates, R. (2012). An initial analysis of communicability evaluation methods through a case study. In CHI EA ’12 CHI ’12 Extended Abstracts on Human Factors in Computing Systems (pp. 2615–2620). New York, NY, USA: ACM. https://doi.org/https://doi.org/10.1145/2212776.2223845
Sieckenius de Souza, C., & Faria Leitão, C. (2009). Semiotic Engineering Methods for Scientific Research in HCI. San Rafael : Morgan & Claypool Publishers.
Arturo Moquillaza.
Magister in Computer Science. Professor at the Universidad San Ignacio de Loyola and Professor at the Pontificia Universidad Católica of Perú. Head of ATM development at BBVA Continental. Member of the UmetSoft research group.
Freddy Paz.
Doctor in Engineering. Full-time professor at the Pontificia Universidad Católica of Perú. Member of the UmetSoft research group.
Fiorella Falconi.
Student of the Master in Computing at the Pontifica Universidad Católica of Perú. Member of the ATM development team of BBVA Continental.
Raysa López.
Systems and computer engineer. Member of the ATM development team of BBVA Continental.