Penny Talk

As the creative director on this project, I managed multiple engineers, internal designers, and external partners, including an animation studio, a toy wood-maker, and a fabrication shop.

We planned 5 major check points along the way (aka “milestones”), which ended with the larger 3-month pilot in the Walnut Creek Capital One Café (Milestone 5). The full project, including the 3-month pilot, took 10 months, from April 2018 to February 2019.

In order to prove the technical feasibility of the project, we created a few preliminary proof of concepts using publicly available software, including FaceRig (shown above) and iOS ARKit.

Simultaneously, we started brainstorming potential characters, each with some sort of “magical” power and a back story. We decided to move forward with DJ Bee, the musically-inclined bee character because of his upbeat, relatable nature (and his lack of ground contact, which decreased a number of technical difficulties). For the detailed creative work, and eventually the matching 3D model and custom animation loops, we had the opportunity to work with Little Fluffy Clouds, an incredible animation shop in Marin.

Before engaging Little Fluffy Clouds to help us create the final high-fidelity 3D model, we created a placeholder bee character. (We wanted to test the waters to make sure the project was worth pursuing further before committing the resources.) Using this initial prototype model of DJ Bee, we created a relatively simple diorama using the iPhoneX’s facial motion capture and keyboard-controlled animation cycles. We got good feedback in initial tests from the Ambassadors (what we call our café employees internally), but we observed (and heard) that they preferred to “talk with their hands.”

Because of the feedback about the desire to gesture, we incorporated hand gesture capabilities using a LeapMotion in our next iteration. Our hypothesis was correct. Although there were a few glitches, especially when Ambassadors made large gestures with their arms, they reported greater expressiveness and ease-of-use using the gesture-controlled version as opposed to the keyboard controls. We also learned that the tech should fall seamlessly into the background — at its heart, Penny Talk is about interactivity and real financial conversations with kids.

For our next milestone, we wanted to test a number of things: 1. If the system could actually be implemented in a real Capital One Café setting, complete with both front-of-house and back-of-house cameras and microphones; 2. If the puppeteer felt more comfortable using Oculus Rift controllers as the gestural system instead of the LeapMotion hand controls; and 3. How a real, unexacting audience responded to the DJ Bee character interaction. The café hardware integration was successful from a technical feasibility standpoint, but we quickly learned that the stability of the large micro-tile screens and the audio levels were problems. We also learned that the Oculus Rig controllers were much more intuitive and less glitchy than the LeapMotion version. Lastly, from the interaction with an audience of 10-year old girls who just happened to be in the café doing calligraphy, we learned that the Penny Talk system needs to be adaptable to different puppeteer personalities.

For all the learnings from this test, watch the Milestone 2 Testing Recap.

It was also around this time that we made significant progress related to the actual system rigs. We went from an extremely heavy helmet + selfie stick combination to a much more professional-looking shoulder rig vest + friction arm — with a few PVC pipe hacks and baby backpack iterations along the way.

Related to the underlying tech of the project, Milestone 2 represented a huge amount of progress for us. After quite a few iterations and lots of testing, we ended up combining facial blend shapes with more natural arm gestures and custom triggers using the Oculus Rift controllers. We still needed to minimize the set-up complexity of the experience and optimize the puppeteer UI and controls, but now the brute of the work fell in the content and end-to-end experience categories.

In order to get a better understanding of the specific age group of kids we were targeting, we did quite a bit of research, including volunteering to teach a JA Workshop for 3rd grade students with café Ambassadors.

We also did a full audit of Turtle Talk to gain a better understanding of an ideal content structure, breakdown, and timeline for kids during interactive shows.

For Milestone 3, we tested our first pass at scripted content with kids in the café (7 kids between the ages of 5 and 11). We also tested the transition to a hands-on craft activity after the show — in this case, we taught the kids how to make an origami bee. One Ambassador performed as the puppeteer from a back room while one Ambassador helped facilitate the experience from the audience-side. We learned that the kids liked interacting with DJ Bee, but even with snazzy animations and music, the tech only goes so far. In the end, the conversation and the story are what keep the kids engaged. We also learned that puppeteers need active talking points during the show while performing (and not just a script to read beforehand) and that facilitators need to play a more active role during the show.

For the very last milestone before the three month pilot, we tested the end-to-end experience, including guided content during the DJ Bee interaction and a hands-on craft activity involving the making of an origami “money box” (aka piggy bank), in a neutral setting. In particular, we tested two different groups of 10 kids ranging between 7 and 12 years old. As you’ll see in the trailer video above, the kids were impressed with the magical interaction component and had a lot of fun. We also learned a lot, so the test was a big success.

As fas as content goes, we learned that that there’s room to incorporate more money-related lessons/content. We also learned an important lesson about the tricky balancing act of improv: It’s important to allow for a bit of improv on the Ambassador side, but we need to make sure they hit certain topics related to the episode content so that the flow makes sense. Regarding the Ambassador roles, this test confirmed that the DJ Bee show is a team effort. The facilitator’s collaboration is equally important to the success of the experience as the puppeteer’s.

To collect some quantitative metrics, we gave the parents a brief survey to complete while the kids worked on their origami money boxes. When we asked them which three words they would use to describe the show, “fun” and “interactive” were the most popular (n=11 and n=9, respectively). As far as the show length, Goldilocks herself couldn’t have been more pleased. Most parents thought the show length was just right, with a few others wishing it was a bit longer. We also asked the parents how likely they would be to bring their kid back to an event like this, and the majority responded positively, giving us an NPS score of 58 right after the event and an NPS score of 30 in the post-survey one week later.

Watch the full Session 1 interaction with DJ Bee here.

The kids and the Ambassadors were our main focus for the 3-month Milestone 5 pilot in the Walnut Creek Café, with a secondary focus on the parents. For the kids, we wanted to answer the following two questions: 1. Can we get kids to engage on financial topics in a café space? and 2. Is the experience captivating and relevant? For the Ambassadors, we wanted to learn the following: 1. Are Ambassadors inspired and motivated to perform? and 2. How can we minimize the impact of set-up and preparation?

Although the parents weren’t our primary focus, we still wanted to learn a few things from them. After all, they’re the ones who will be deciding whether or not they bring their kids to the café for this experience. They have a strong financial impact on their kids, so we wanted to start to answer the following two questions: 1. Can we inspire parents to engage with the Capital One brand? and 2. Can we help give families the confidence to talk about money?

For the Milestone 5 pilot, we tested three different episode + craft activity combinations each weekend for a month at a time. For the content, we relied on the monthly marketing themes that had already been set for the Walnut Creek café, as well as the time of year, for inspiration.

For example, Episode 1, called DJ Bee Gives Back, was created to align with the November theme of Giving Back — the premise being that DJ Bee needs help brainstorming ways to give back to the community. Episode 2, called DJ Bee’s Winter Break, focuses on his appreciation for his family at the end of the year. During the interactive portion of the show, the kids help him plan a special winter break trip to celebrate his family’s hard work. Then, after the interaction with DJ Bee, they got to decorate a winter picture-frame.

I’m proud to say that we collaborated with MakeNPlay, a local small business owner that specializes in wood toys, to create the craft activities for this pilot. The three craft activities included a honey-comb-inspired money box, a winter picture frame, and a SMART goal framework, which can be hung on a wall to track specific, measurable progress towards personalized goals. Ismail, the founder of MakeNPlay, actually incorporated similar activities into both his physical retail and his online Etsy store. He says these Penny Talk-inspired kits, especially the picture frame, have significantly increased his store’s revenue.

Throughout the pilot, we were constantly making updates to the set-up process and the puppeteer UI. By the end, with some manufacturing help from SMA Events, a fabrication shop based in Oakland, we had a fully integrated rack on both the facilitator and puppeteer side. We started off with two suitcases full of gear for the first week of the pilot, so the Ambassador’s were thrilled about these updates to make the set-up process more seamless.

Due to the test-like nature of the pilot, we weren’t allowed to market the experience. That was no issue though — the Ambassadors were so excited to spread the word about the show that over 60 people got to experience Penny Talk in the Walnut Creek café over the course of the three-month pilot. The kid to parent/guardian ratio was about equal with some one parent, multiple kid combos and some one kid, multiple guardian combos. Some of the families came from as far as an hour away to visit the Walnut Creek café. The show was flexible enough to allow Ambassadors to adapt to different age ranges of kids as well as late-comers.

From the 28 kids, we learned that the experience is quite magical and that the hands-on craft activities are a hit. The kids stayed engaged for the full-hour (which we know is impressive from our experience volunteering with JA) and we heard from the Ambassadors that many of the kids even talked about their experience later at home!

From the 26 parents who experienced the show, we learned that they like to participate just as much as the kids. They prompted their kids during the interaction with DJ Bee and gave hands-on help during the craft activity portion. Although no products or sales pitches were mentioned during Penny Talk, we saw parents opening Kids Savings Account after the show. The parents also asked about other activities for kids in the café — there is definitely a desire for more kid-related programming.

We also learned a lot from the 6 Ambassadors who performed as both the puppeteer and the facilitator for the show. They actively gave us feedback throughout the three month pilot, both in the form of quantitative follow-up surveys after each show as well as genuine conversations.

We saw that the Ambassadors love getting creative. Both the puppeteers and the facilitators enjoyed putting their own spin on the show, resulting in smiles and laughter from kids and parents alike. The Ambassadors loved being part of the process and appreciated having a say in the show’s improvements. In fact, it was actually one of the Ambassadors who introduced us to the MakeNPlay vendor and took the initiative to order cookies in the shape of a bee for kids who attended the show.

From the surveys we sent out after each show, 100% of the Ambassadors said they would both volunteer to participate in another show and recommend being a puppeteer or facilitator to other Ambassadors in the café. The scale of how comfortable they felt improvising was generally positive with all answers falling in between 7 and 10 on a scale of 1 to 10. With the frequent changes to the set-up process, it’s no surprise that the question about the set-up process difficulty has a much larger range of answers.

Overall, we received extremely positive feedback about the experience from them. Mostly, they appreciated the opportunity to be involved in such an impactful project with kids, who are an extremely moldable (and cute!) group. The technology gave the Ambassadors an avenue to connect with kids that was exciting and fulfilling.

One of the things we asked the Ambassadors in our Milestone 5 wrap-up interviews was “What three words would you use to describe the Penny Talk experience?” Watch the video recap above to hear it in their own words, but some of the words that came up were “creative,” “innovative,” and “impactful.”

We also got good feedback around things that still need improvement. Because the Penny Talk experience needs two Ambassadors — one on the front-end and one one the back-end for about an hour and a half (to set up, actually perform the show, and then clean up) — and a certain number of Ambassadors have to be on the café floor at all times, there was a unanimous pain point surrounding scheduling. We also heard that the grass-roots, word-of-mouth marketing was sometimes difficult without the use of photo/video materials for the Ambassador to refer to. It turns out that explaining Penny Talk is difficult with visuals.

During our last month of Penny Talk shows in the Walnut Creek Café, we started preparing to hand the project over to another internal team to scale it up to other Capital One Cafés. In particular, we worked with Cinema-Suites to help us create both an official launch screen and an episode creator, so that the system would stand-alone and a future team would be able to create their own show content, respectively. The episode creator included a selection of matte background images with associated music as well as text fields for puppeteer hint text and other script components.

We have a number of utility patents pending right now both for the over-all system and some specific unique technical components.