Recommenders in Ecommerce
Introduction
The presence of recommenders has become ubiquitous in daily life over the past decade or more, from choosing what to stream on video, which music or podcast to listen to, who to pick us up and take us where we need to go, to when is the best time to leave to beat the traffic or other conditions. Experts agree that improved or hyper-personalisation offers substantial value to both consumers and businesses, with 71% of McKinsey (2023) survey participants stating they expect personalisation, Accenture (2024) finding that 75% of surveyed consumers want still quicker and simpler ways to identify products that meet their needs, and Bain & Company (Baudry et al., 2024) recommending that brands combine organisational readiness, content creation, analytics, and appropriate tools towards this end, so it is no surprise that brands are doing their best to make use of today’s recommenders in meeting and exceeding those standards.
The benefits of recommenders can include increased revenue, enhanced user experience, higher conversion rates, increased average order value, and improved customer loyalty (ibid.). While on the downside, there are risks to be managed and drawbacks to be avoided – among them privacy, data quality, unintended effects, discrimination, and lack of transparency - and at the same time, opportunities to stay ahead of, with advances being made in AI at a pace that has seen the technology progress from generative AI assistants to the advent of AI agents in only two years. The recommendation system is a technology that we’re all familiar with in its current state today, using it so often that we hardly notice it is there, and perhaps know less about its origins to appreciate how far it has come. This short article seeks to shed a little light on the long story of its development and those who have contributed to where it is today.
Beginnings
Whilst the human search for good advice reaches back to ancient times (Schrage, 2022), the use of technology to solve this problem is said to have started 45 years ago in 1979 (see Benedetto, 2023) when Elaine Rich, later to become Director of the Microelectronics and Computer Technology Corporation (MCC) Artificial Intelligence Laboratory, author of two books on AI, Distinguished Senior Lecturer at The University of Texas at Austin, and now retired, completed her doctoral dissertation at Carnegie-Mellon University on a system called Grundy, that she designed to help users find books they might like and address “the problems that must be considered if computers are going to treat their users as individuals with distinct personalities, goals, and so forth” (Rich, 1979, p.1), demonstrating “that stereotypes (models of groups of users who share common interests or characteristics) could be effectively exploited by a system [Grundy] that gives advice to people on things they might like” (ibid.). Hence the origins of the recommendation engine stem from a drive to improve the human-computer interaction by personalising the process and outcomes that has propelled its evolution over the ensuing years to today.
Development
It was over a decade before the next major milestones were recorded in the development of the recommender system. In the early days of the Internet, three academic research teams applied the technology in electronic mail, news, and music at Xerox Palo Alto Research Centre (PARC), in a joint collaboration between MIT and University of Minnesota, and at the MIT Media Lab, respectively. While Rich’s Grundy had to operate within the constraints of early computing using constructed stereotypes upon which to formulate recommendations, this second generation of efforts had access to albeit limited data available from the dial-up bulletin board systems where users would share their opinions, views, or other descriptive, often subjective, information about the relevant item.
In 1992, the concept of collaborative filtering is said to have been introduced (Jannach et al., 2021) with the experimental email filtering system called Tapestry (Goldberg et al., 1992), inspired by the increasing use of ‘electronic mail’, created by a team of researchers at Xerox Palo Alto Research Centre led by computer scientist, David Goldberg. The system was designed to process any incoming stream of electronic documents, including email and news stories or articles, and worked somewhat by allowing the user to define rules or criteria for email they wished to receive, which were matched to descriptions left by other users (for a much clearer explanation see ibid.).
Two years later a research team formed by academics from the MIT Centre for Coordination Science – studying how coordination occurs across computer, human, and market systems – and University of Minnesota Department of Computer Science, led by Paul Resnick (Resnick et al., 1994), sought to automate the collaborative filtering process of Tapestry with a news recommender called GroupLens. In a reflection on the comprehensive history of the recommender, Jannach et al. (2021) describe GroupLens as one of the first systems based on explicit ratings of a user community and employing ML to determine the likelihood of a user wanting to view particular items.
While all this was happening, then MIT Associate Professor Pattie Maes had founded and directed the Software Agents Group at the MIT Media Lab from 1990 (Britannica Money, 2024), described as “semi-intelligent computer programs which assist a user with the overload of information and the complexity of the online world” (Maes, 1996). Maes led research projects in autonomous agents and recommender systems which produced successful products of the time such as the 1992 HOMR – Helpful Online Music Recommendations – prototype which later became the Ringo email-based music recommender in 1994 (see Schrage, 2022, Johnson, 2022, and Kirsner, 1998), and Firefly, the first commercial, peer-to-peer music recommender in 1995. The latter was also among the first to incorporate user privacy and identity tools, and sold to Microsoft in 1998, primarily for its identity technology (ibid.). For a glimpse at how the work of Pattie Maes and the MIT Media Lab team improved the consumer experience, a short piece written for Wired in 1996 by the Lab’s co-founder and chairman Emeritus, Nicholas Negroponte, ‘Electronic Word of Mouth’, is insightful and illuminating.
State of the Art
In 1994 as the GroupLens paper was published and the MIT Media Lab completed Ringo, the first secure online transaction took place – apparently a Sting CD from NetMarket (see Fast Company, 2015) – and modern ecommerce was born (Pew Research Centre, 2014), bringing a third generation of recommender systems. Though academic research in the domain continues today, with some of the original researchers still practising, such as now Professor Pattie Maes of Media Arts and Sciences at the MIT Media Lab (MIT Media Lab, 2024), and the ongoing GroupLens research with its namesake program at University of Minnesota, ecommerce startups at the time and thereafter picked up the recommender ball and have been running with it since, with the benefit of scale, big data, and the resources and motivation to invest in creating the best.
Amazon introduced its first recommender system in 1998 (Smith and Linden, 2017), four years after its launch as an online bookstore, and almost twenty years after Grundy had become the first ever book recommender. The Amazon recommender system applied item-based collaborative filtering to allow the user to discover items they might like but had not yet seen, with the added ability to update immediately based on new information about a user, and an early demonstration of transparency by being able to explain its recommendations to the user (ibid.). Twenty-six years later, Amazon has released Rufus, its generative AI-powered expert shopping assistant which can make recommendations based on its training in Amazon’s product catalogue, customer reviews, community Q&As, and other web information to answer queries about shopping needs, products, and comparisons (Mehta and Chilimbi, 2024).
Other ecommerce platforms that launched in the late 1990s to 2010s have followed Amazon’s lead with AIML-driven recommenders to assist marketers and their customers, including eBay, Alibaba, Shopify, Salesforce, and Adobe. Shopify was one of the first to adopt the advantages of generative AI, launching Shopify Magic in 2023 (Shopify, 2023) to improve marketing productivity, customer experience, and results, and Sidekick in 2024 (see Peters, 2024), its AI assistant for marketers. Salesforce entered the arena in 2016 with its acquisition of Demandware (Salesforce, 2016), transforming it into Salesforce Commerce Cloud, and in 2024 launching Agentforce (Salesforce, 2024), allowing brands to create their own AI agents, freeing up human capital to focus on personalisation and relationship-building. Similarly, though Adobe was a latecomer to ecommerce in 2018 with its acquisition of Magento (Adobe, 2018), it has been a fast-mover in finding a competitive advantage with gen AI through the launch of Adobe Sensei (Adobe, 2023) and its incorporation into Adobe Commerce, such as through Product Recommendations (Adobe, 2024).
From another angle, generative AI platforms themselves are getting involved in ecommerce recommender systems, though each in quite different ways. OpenAI assisted Klarna to become the first European company and first fintech firm to launch a ChatGPT plugin in 2023, which has since been developed into a chat-based shopping assistant (OpenAI, 2024; Klarna, 2024). Anthropic launched ‘tool use’ in 2024 (Robison, 2024; Anthropic, 2024), allowing anyone to connect its AI chatbot Claude to an external API for purposes such as creating personalised product recommendations, answering customer enquiries, or for a consumer to create their own personalised bot to purchase products for them. More recently, on 18 November Perplexity launched its own AI-powered shopping assistant (Perplexity, 2024), which allows consumers to search by image, research, and purchase products directly from the Perplexity website or app.
Possibilities
Each of these efforts takes us another step closer to having our own AI shopping agents - our own customised ‘recommenders’ - that we could interact with by voice, when we needed to interact with it at all, as it would already be familiar with the preferences, tastes, and needs that we had allowed it to access through our data, and it could not only anticipate our routine purchases and activities and how we normally conduct them, but take action with our permission to complete the purchase itself and carry out any follow-ups, complaints, or returns on our behalf. We could be less reliant on the digital interfaces that have become so familiar over the past three decades as they’re slowly replaced by an invisible interface with autonomous AI that is, with our authority and within limits we set, able to act for us.
Bain Capital Ventures (BCV) refers to this prospect as the "Agentic Commerce Era" (Friend, 2024), highlighting a period that could be as disruptive as the beginnings of ecommerce and mobile shopping were, where the brands who stay ahead of the increasingly acceleratingly pace of technological development, creative ideas, and time-to-market will be best placed to gain a competitive advantage, shaking out those who lag behind. It is suggested that brands will need to invest more in new capabilities, technologies, and infrastructure to optimise the benefits and manage the risks of implementing AI agents (ibid.), and it is those with the agility to act quickly who will come out ahead. In this environment, it could be interesting to see how the original trailblazer, Amazon, uses the unique position it has since built across ecommerce, cloud computing, and smart homes to innovate and perhaps again lead ecommerce into a new era.
Summary
As the recommender system rapidly approaches 50 years of development, we can see that vast progress has been made by virtue of the inventive researchers dedicated to improving human-computer interaction, creative entrepreneurs in the quest to innovate, and the scientists and engineers developing the increasingly sophisticated technologies at their disposal. Yet, there are still five more years to reach this landmark occasion, and still much room for the particular design and practical implementation of the recommender system to be improved in ecommerce – from suggestions that miss the mark by such a distance it leaves consumers questioning how well the brand really knows them, to recommendations that make the consumer uncomfortable and worried for their data privacy.
Here, gen AI and AI agents hold much potential in reaching a new level of personalisation that can more fully comprehend the context and nuances of consumer needs and the circumstances surrounding them through conversation and collaboration, simultaneously reducing the cognitive workload and effort involved in information-seeking, decision-making, and action-taking, whilst instilling the consumer with increased agency to define and fulfil their preferences more effectively. The key to unlocking this opportunity lies in not only leveraging AI to personalise the human-computer interaction, but to humanise it, creating more empowering partnerships with consumers. With such significant advancements being made in recommenders over the past two years with gen AI, the next five look set to be fascinating.
References
Accenture (2024). The empowered consumer. Retrieved on 29 November 2024 from https://www.accenture.com/au-en/insights/consulting/empowered-consumer.
Adobe (2024). Introduction to Product Recommendations. Adobe, Documentation, Commerce, Product Recommendations Guide. Viewed on 2 December 2024 at https://experienceleague.adobe.com/en/docs/commerce-merchant-services/product-recommendations/overview.
Adobe (2023). Adobe announces new Sensei genAI services to reimagine end-to-end marketing workflows. Adobe, News, 21 March 2023. Viewed on 2 December 2024 at https://news.adobe.com/news/news-details/2023/adobe-announces-new-sensei-genai-services-to-reimagine-end-to-end-marketing-workflows.
Adobe (2018). Magento is now Adobe Commerce. Adobe, Blog, The Latest, 19 June 2018. Viewed on 2 December 2024 at https://business.adobe.com/blog/the-latest/magento-is-now-part-of-adobe.
Anthropic (2024). Claude can now use tools. Anthropic, News, 31 May 2024. Viewed on 2 December 2024 at https://www.anthropic.com/news/tool-use-ga.
Baudry, G., Grudnowski, J., and Pavy, S. (2024). The art and (data) science behind effective customer personalisation. Bain & Company. Viewed on 29 November 2024 at https://www.bain.com/how-we-help/the-art-and-data-science-behind-effective-customer-personalization/.
Benedetto, L. (2023). Recommender systems: benefits and practical guidelines for software professionals. SWForum.eu, May 2023. Viewed on 2 December 2024 at https://swforum.eu/online-sw-forum/software-technology/6/recommender-systems-benefits-and-practical-guidelines.
Britannica Money (2024). Pattie Maes. Britannica Money, Biographies, Entrepreneurs. Viewed on 2 December 2024 at https://www.britannica.com/money/Pattie-Maes.
Fast Company (2015). You’ll never guess what the first thing ever sold on the Internet was. Fast Company, Fast Feed, 26 November 2015. Viewed on 2 December 2024 at https://www.fastcompany.com/3054025/youll-never-guess-what-the-first-thing-ever-sold-on-the-internet-was.
Friend, S. (2024). The dawn of the agentic commerce era: Putting Gen AI to work for shoppers. Bain Capital Ventures. Viewed on 29 November 2024 at https://baincapitalventures.com/insight/the-dawn-of-the-agentic-commerce-era/.
Goldberg, D., Nichols, D., Oki, M., and Terry, D. (1992). Using collaborative filtering to weave an information tapestry. Communications of the ACM 35(12): 61-70. Retrieved on 2 December 2024 from https://www.scirp.org/reference/referencespapers?referenceid=1776081.
Jannach, D., Pu, P., Ricci, F., and Zanker, M. (2021). Recommender systems: Past, present, future. AI Magazine, 42(3), 3-6. Retrieved on 29 November 2024 from https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/18139.
Johnson, S. (2022). Intelligent agent: How Pattie Maes almost invented social media. Netguru, Hidden Heroes. Viewed on 2 December 2024 at https://hiddenheroes.netguru.com/pattie-maes.
Kirsner, S. (1998). Firefly: From the Media Lab to Microsoft. Wired, Business, 9 April 1998. Viewed on 2 December 2024 at https://www.wired.com/1998/04/firefly-from-the-media-lab-to-microsoft/.
Klarna (2024). Shopping made smarter: Klarna adds more AI features to its assistant powered by OpenAI. Klarna, 19 September 2024. Viewed on 2 December 2024 at https://www.klarna.com/international/press/shopping-made-smarter-klarna-adds-more-ai-features-to-its-assistant-powered-by-openai/.
McKinsey & Company (2023). What is personalisation? McKinsey Explainers, 30 May 2023. Viewed on 29 November 2024 at https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-personalization.
Mehta, R. and Chilimbi, T. (2024). Amazon announces Rufus, a new generative AI-powered conversational shopping experience. Amazon, News, Retail. Viewed on 29 November 2024 at https://www.aboutamazon.com/news/retail/amazon-rufus.
MIT Media Lab (2024). Pattie Maes, MIT Media Lab, People. Viewed on 2 December 2024 at https://www.media.mit.edu/people/pattie/overview/.
Negroponte, N. (1996). Electronic word of mouth. Wired, Message 40, 1 October 1996. Viewed on 2 December 2024 at https://web.media.mit.edu/~nicholas/Wired/WIRED4-10.html.
OpenAI (2024). Klarna’s AI assistant does the work of 700 full-time agents. OpenAI, Customer Stories. Viewed on 2 December 2024 at https://openai.com/index/klarna/.
Perplexity (2024). Shop like a Pro: Perplexity’s new AI-powered shopping assistant. Perplexity, Hub, Blog, 18 November 2024.
Peters, J. (2024). Shopify’s AI ‘Sidekick’ chatbot for merchants is now in early access. The Verge, 24 June 2024. Viewed on 2 December 2024 at https://www.theverge.com/2024/6/24/24183504/shopify-ai-sidekick-chatbot-merchants-summer-edition.
Pew Research Centre (2014). World Wide Web Timeline. Viewed on 29 November 2024 at https://www.pewresearch.org/internet/2014/03/11/world-wide-web-timeline/.
Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., and Riedl, J. (1994). Grouplens: An open architecture for collaborative filtering of Netnews. In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, CSCW ’94, 175-86. Retrieved on 2 December 2024 from https://dl.acm.org/doi/10.1145/192844.192905.
Rich, E. (1979). User modelling via stereotypes. Cognitive Science 3, 329-354. Retrieved on 29 November 2024 from https://www.cs.utexas.edu/~ear/.
Rich, E. (1983). Users are individuals: Individualising user models. International Journal of Man-Machine Studies (1983) 18, 199-214. Retrieved on 29 November 2024 from The University of Texas at Austin, https://www.cs.utexas.edu/~ear/.
Robison, K. (2024). Anthropic’s AI now lets you create bots to work for you. The Verge, 31 May 2024. Viewed on 2 December 2024 at https://www.theverge.com/2024/5/30/24167231/anthropic-claude-ai-assistant-automate-tasks
Salesforce (2024). Salesforce unveils Agentforce – what AI was meant to be. Viewed on 2 December 2024 at https://www.salesforce.com/au/news/press-releases/2024/09/12/agentforce-announcement/.
Salesforce (2016). Salesforce completes acquisition of Demandware. Salesforce, News & Insights. Viewed on 2 December 2024 at https://www.salesforce.com/news/press-releases/2016/07/11/salesforce-completes-acquisition-of-demandware/.
Schrage, M. (2022). History’s great recommenders: From ancient times to tomorrow. The MIT Press Reader, MIT Press, 5 December 2022. Viewed on 29 November 2024 at https://thereader.mitpress.mit.edu/historys-great-recommenders-from-ancient-times-to-tomorrow/.
Shopify (2023). Introducing AI-generated product descriptions powered by Shopify Magic. Shopify, Blog, Updates, 20 April 2023. Viewed on 2 December 2024 at https://www.shopify.com/au/blog/ai-product-descriptions.
Smith, B. and Linden, G. (2017). Two decades of recommender systems at Amazon.com. IEEE Internet Computing Vol. 21 Issue 3, pp. 12-18. Retrieved on 29 November 2024 from https://ieeexplore.ieee.org/document/7927889.
