Scaled up personalization that hits the mark

Back to Blogs

89% of leaders believe personalization is crucial to their business' success in the next three years. Twilio Segment, The State of Personalization Report 2024

In addition to customer experience, how does personalization integrate with other touchpoints, such as commerce and marketing? What should you, as a data and technology leader, be thinking about?

Methods of personalization

Personalization is all about understanding your customers better and using those insights to deliver intelligent experiences that resonate. Data helps you understand more about a customer as an individual. When we're talking about a personalized system, we're talking about a solution with some kind of intelligence working behind the scenes. There are several ways to approach personalization.

Rule-based personalization

This is the most straightforward approach to personalization. Simply put, rules are set up based on the customer's past behavior. For example: Give me all customers who haven't been active in 30 days and who have previously purchased peaches.

With this technique, personalization rules are easy to apply; it also gives your domain experts control over the results. However, rule-based personalization tends to provide diminishing returns. As the number of rules you're applying grows, it becomes incredibly cumbersome to manage them, or to know which of the rules is actually making a measurable difference to customer behavior.

An effective use of rule-based personalization is as a post-processing step- after machine learning has been used to automatically determine interesting aspects of customer behavior.

Segment-based personalization

This technique involves applying rules and machine learning to customer behavior in order to group or segment similar customers. A classic technique called RFM (recency, frequency, monetization) puts customers into segments like "high value, highly engaged", or "low value, highly engaged". You can run multiple segmentations side by side and then combine them to achieve an in-depth view of your customers - a combination like "high value customers, who have low purchase frequency, who are interested in this niche product" is easily created by combining multiple segments together.

The advantage of segments is that they can be well understood by different groups within your organization. Perhaps your company is trying to grow a particular target segment, so multiple lines of business (think marketing, commerce, sales) are interested in how those users behave. Since these groups are looking at those segments daily, they're well verified.

The downside of leaning too heavily on segment-based personalization is that you're not really gaining an understanding of customers as individuals. To do so, apply more machine learning and AI to drill into more available data.

Embedding-based personalization

This is the technique that most modern personalization systems use. An embedding is a vector- a long list of numbers- that describes an individual customer. By using a machine learning technique, such as collaborative filtering, historic customer behaviors are reviewed and all of those behaviors are encoded into a unique embedding for each customer.

The advantage of using embeddings is that vectors are easy for computers to store and work with, so having an embedding stored for each customer makes it easier for your organization to create a personalized system.

The downside of embeddings is that they're not easy for a human to interpret. Reading large lists of numbers is easy for a computer, but not for people. Often, embeddings are used to create the heart of the personalization system, then techniques like rule-based personalization are applied on top of those results to give control to domain experts- while still retaining the power of machine learning.

How embeddings were used to construct recommendations for YouTube : "Deep Neural Networks for YouTube Recommendations"

The role of generative AI in personalization

We now have enough experience with generative AI to know how it can be used to create impactful personalized experiences. On the other hand, we also realize that genAI has some "rough edges".

Let's take a look at those rough edges first:

• Running a genAI system at scale is difficult. They're prone to hallucinating or even inventing information.
• A scaled up genAI system is difficult to maintain and expensive to operate.
• Exposing a genAI model directly to customers is risky. For example, this meal planner created poisonous recipes!

Rather than directly exposing a genAI model to customers, organizations should take a more risk-managed approach. In the current generation of generative AI models, there has been success with:

• Processing large volumes of unstructured customer data: GenAI models are great at processing unstructured data (e.g. clickstream data, user-generated content such as free text). Processing this data in the backend can enrich other structured data sources.
• Creating more content from which to personalize: There isn't much use in personalization if you don't have a lot of content to recommend to users. Organizations are increasingly using genAI to expand the content they have, and use it to deliver deeper experiences based on personalization.
• Applications like search: GenAI is great at knowledge management, and creating a more interesting and useful user experience for applications like search or information retrieval.

Fair, human-centered systems

If personalization is treated as purely a technical and data challenge, it will fail. After all, we're personalizing to gain a better understanding of what humans want and need in order to build better products for them. An excellent personalized experience only happens when humans are at the center of the solution, when the need for relevant content is balanced with the need for novelty and surprise. Excellent personalized experiences are designed to expand the customer's horizons rather than recommending what they're accustomed to selecting.

Fairness is all about creating a system that doesn't entrench previous behaviors, but makes sure that the system corrects for algorithmic bias. Multidisciplinary teams are key to developing this more strategic view; it's a far better approach than people working in silos.

People, process, technology: how to do personalization at scale

We've talked about how you can create personalization from an algorithmic perspective. Now, as a leader, what should you be thinking about in terms of your people, processes, and technologies? What types of investment and organizational support should you consider?

People in personalization: Agile, insight generation, team topologies

Agile teams, test and learn: An agile methodology is highly effective in building personalized products. Start with simple personalization experiments, learning from each one, then fine-tune your products as you learn more. A cross-disciplinary team of product, engineers, data scientists, user experience and design works incredibly well.

Insight generation: Some individuals in your organization, such as data scientists or analysts, should be exploring signals in the data and finding what's useful in order to develop a better understanding of your customers, so that they can share these insights with other internal teams.

Team topologies: This is an excellent framework for thinking about how teams work together. In a team topologies model, how might your teams collaborate to develop personalized products?

• Stream aligned team - These people are responsible for understanding the customers, developing algorithmic approaches for personalization, and building products around them. This team leverages the capabilities from the platform team.
• Platform team - Personalization is a data-heavy effort. A platform team provides the capabilities for ingesting, processing, and serving data- reducing the cognitive load for the stream aligned team.
• Complex subsystem team - The deep insights that create a personalized view of the customer need to come from a single place. This team indexes heavily on data science and user research.

Process in personalization: Strategy and governance

Strategy: Start simple - Organizations often get lost by trying to create a complete and strategic view of what personalization means, and then start work. But companies that take an agile approach are more successful. They have a big vision for what personalization means, know they are going to iterate to get there- and learn along the way.

Governance - It's essential to have guardrails in place to ensure that your organization takes a human-centered approach to personalization. Make sure that all stakeholders are represented and are working together to achieve the same goal. Governance prevents different parts of the organization from having disparate views of the customer.

Data and technology in personalization

There are a few technology investments that will streamline the personalization journey.

Data engineering: High quality and accessible data is necessary for all teams involved in personalization. Investing in your data engineering plumbing ensures that your team won't have to repeat the work of cleaning and accessing data each time. This may require investing in your customer data platform (CDP) to add more data to it, or it may be extracting data from your CDP into your data platform.

There's a lot of talk about requiring a 360-degree view of the customer- a situation where all interactions are logged and attributed to a particular customer. But a full 360-degree view of the customer is often an unnecessarily cumbersome task. Some data points may be ridiculously hard to access, or perhaps activities aren't tracked digitally, or data is locked away in a third party system. It's not a bad idea to get the ball rolling by starting with a less complete view of the customer, and then add in more data points as you progress.

Channels like marketing are plugged in: Your marketing channels are incredibly valuable, both as a source of customer data, and also as a personalization channel. Ensure that the marketing channels are well integrated with your data ecosystem.

Shared infrastructure: Some shared infrastructure components are very useful, and investing in these will give each team building personalization a "batteries included" experience:

- A feature store

• Is a low latency read/write database that stores features about customers derived from your personalization algorithms. For example, customer segments, particular interest groups, etc.
• These features are then requested every time a customer interacts with your product. Customer features from the feature store are used as part of the product experience.
• It is important to apply governance to your feature store in order to avoid including unencrypted personal information and to ensure that features are documented and retired when they're no longer useful.

- A way to deploy data products through API

• A cookie cutter approach, so your APIs are easy to deploy.
• Ensure that APIs are scalable, have monitoring/alerting/observability baked in, CI/CD and other good deployment practices.
• If you have a machine learning backed API, a good deployment strategy is to have a separate train/test/evaluate pipeline to retrain and update the model; so when you deploy the API you are only changing the trained model artifact.

You will want to take a site reliability engineer (SRE) approach; these are core infrastructure elements, so it is important to respond when they are laggy, not caching well, hitting hot shards in your database, etc.

Sandbox environment for data scientists: As data scientists will be continuously experimenting and discovering new user features, they will appreciate having a sandbox environment to conduct exploratory work. Some features of this sandbox environment:

• Should have access to real and anonymized data.
• Should have governance processes in place to ensure costs are monitored, long running processes are terminated, and nothing is built that you need to depend on.
• Should have a well understood graduation path to production environments.

Testing infrastructure such as A/B testing: It is very important that teams are able to learn about and quantify the impact of their personalization work on the user experience. The personalized experience can be the most impactful A/B test that an organization has ever run.

Get going with your personalization strategy

Kin + Carta is making it easier to activate on your commerce data to enable hyper-personalization. Our Integrated Commerce Network (ICN) is a curated group of software partners that enables transformation across commerce, marketing, and customer experience with pre-integrated best-in-class data, commerce and personalization capabilities.

Ready to accelerate your commerce data?

Learn more

Share this article

More like this

Show me all