Every product manager talks about data-driven product management but what is the real explanation of it?
Product decisions are used to be based on product managers/owners and C-Level executives’ desires and instincts. What is the main driver of these instincts; customer feedbacks, competitive market intelligence or digital analytics results? The answer should be all of them. In this article, I will try to explain crucial data sources and which metrics should be considered in these data sources to make right product decisions.
Successful products belong to customers more than product managers. But it doesn’t mean product managers should only rely on customers’ needs and demands. Sometimes even customers don’t know what they want and need. Product managers must validate their customer feedbacks with different data sources.
As stated in the chart, Product Management should consider the data from Market Researches, CMI (Competitor Market Intelligence), Internal Feedbacks, Digital Analytics and Customers’ Feedbacks to make data-driven decisions and create a value-added information. Sounds easy but assessing the value of the information and tying them to product roadmap needs an effort. Let’s deep dive into them one by one, and I will try to explain how I am using this model in my product management efforts.
What is the probability of heads or tails when we flip a coin? 50%? Unfortunately, no! If we consider all the variables in the equation such as the initial force applied when flipping, gravity, air resistance, etc. there is no probability of the coin. It is same in product management and positioning too. If product managers can consider right market metrics and trends by proper market researches, and useful customer feedbacks, there is no probability to build a successful product.
Forrester provides Reports, Playbooks, Blogs, Data and Events to spread out their research outputs. Especially market specific Forrester Wave Reports are useful for market researchers. Forrester Wave researches provide the same outline in all researches for standardization. This standardization allows researchers to combine multiple research easily and reach useful insights. Usually, their Wave Report outline is;
- Key Takeaways
- Overview for Market Professionals
- Evaluation Overview of Key Players
- Positioning Graph
- Strong Performers
- Evaluation Table
- Current Offerings
- Market Presence
- Vendor Profiles
However, Gartner uses “Magic Quadrant” methodology in their researches to show the market positioning of vendors. They assess key vendors by the ability to execute and completeness of vision. They categorize them as Leaders, Challengers, Niche Players and Visionaries. Usually, their report outline is;
- Market Definition / Description
- Magic Quadrant
- Vendor Strengths and Cautions
- Vendors Added and Dropped
- Inclusion and Exclusion Criteria
- Evaluation Criteria
- Niche Players
- Market Overview
These researches show strengths and the weak points of the vendors in the market and help product managers to validate their positioning. Sometimes niche products and ideas struggle to find a market or they may be in need to validate their vision and roadmap in accordance market trends. Market researches help product managers to validate their positioning and the product vision to give data-driven decisions.
Competitor Market Intelligence
This term is used to be named as a function in enterprise companies. CMI departments constantly check other competitors, monitor their contents, annual reports and request a demo to understand competitor value propositions and differentiation. The easiest way of making CMI data available for product roadmap is comparison methodology. The critical point of this methodology is comparing your product as a value proposition and comparing your product as a feature set.
Value proposition comparison usually comes from vendor selection criteria in the market. For instance, let’s consider “Implementation,” “Timeliness,” “Customization,” “Accuracy” and “User Interface” as a main vendor selection criteria for analytics market and define KPIs of these value propositions to be able to evaluate between 100-0 pts. Regarding this evaluation and comparison, you’ll have a pentagon graph to visualize each dimension of your company and your competitors such as an example in below.
Understanding strengths and weaknesses of the value propositions, these pentagons can be used as an area graph and see the comparison by overlapping them.
Feature set comparison is used to be based on product use cases and feature offerings. You compare features or use cases which represent sub-KPIs of value propositions that we listed in the previous paragraph. The general approach to evaluate and compare feature sets or use cases is giving a point them between 1-5 or 0-100 pts.
I’d like to take attention to product positioning at this point. A successful product doesn’t mean a product which has a full advantage on all value propositions and features set comparisons. Your strength areas must be compatible with your product positioning.
For instance, here is a sample table to compare feature sets and validate the positioning of an imaginary company in analytics market:
Comparing value propositions and feature sets help product managers to improve and prioritize their roadmap for competitive advantages. If it requires an improvement for current product, product managers usually tie these insights to their relevant product concepts. If it requires a new feature, product managers build a concept around of it.
Defining right unique selling points (USPs) and building experience around of those USPs’ use cases is crucial for successful product management. The best way to gather these kinds of insights is internal feedbacks especially from “Pre-Sales,” “Sales”, “Consulting” and “IT” departments. These departments are important stakeholders in product management decision making.
Every product has some “AHA!” moments to hook the potential customers. Pre-sales is an important stakeholder to understand which use cases are the most important for customers and what customers are thinking about the product when they first see it.
Covering important use cases mostly doesn’t mean that product is good enough to satisfy customers in those use cases. “Sales” and “Consulting” are important stakeholders to understand how good product’s feature sets to answer real needs and wishes in those use cases. For instance, a customer wants to go from A to B in the product, if the product takes the customer to a lot of mandatory destinations before B, we can’t consider the experience is covering that use case perfectly even the product covers that use case. “Sales” and “Consulting” are critical stakeholders to discover this kind of bottlenecks.
Ease of integration and use, onboarding experience and documentation are another topic that product managers must consider. “Consulting” is the key stakeholder in this kind of inputs. Maximizing the billable time of consulting, the product should have a good onboarding experience and documentation. This part is important to keep COGS (Cost of Goods Sold) at a reasonable level.
Another important stakeholder of “Internal Feedbacks” is IT. Understanding the challenges behind of the concepts, estimated workforce and investment for the roadmap items are important to be able to calculate more accurate business values before making a product decision. Also synchronizing with product owners and IT leads may helpful to see and evaluate product team solutions in some different perspectives.
To quote W. Edwards Deming, “We trust in God; all others bring data.”. Regarding pixels and tag integrations, online solutions gather almost all device, user, and action data. Right event schema and analysis structure is the key point to drill down and reach to the right data for understanding customer behaviors and their pain points easily.
I would like to mention two important topics about digital analytics in this article. Please find detailed information from my “7 Predictions for the Analytics Market” article and my next posts.
The first topic is the importance of combining all your data into one platform. Product managers used to reach combined data by data warehouses, data management platforms or customer intelligence solutions. Nowadays, businesses gather big amount of data from various data sources and internal/external services such as Magento, MailChimp, Zendesk, Salesforce, Olark, etc. Combining these platforms’ data into one environment is important to be able to do cross-tool-analyses. For instance, we should combine digital analytics and email service data and match them with unique identifiers to reach how many men appeal buyers opened our “New Season Men Appeal Products” newsletter. Then maybe we can target them with retargeting campaigns by their cookie data and label returners as a “potential new season product buyers” in our database automatically by connecting our advertisement and CRM services to our digital analytics platform.
The second topic is using proper analyses and monitoring right metrics for the desired outputs to give data-driven decisions. Digital analytics solutions offer a bunch of analyses options. I will deep dive to all these analyses in my next blog posts, but I will mention just a few important analyses and metrics in this article.
Funnels: Funnels are one of the best ways to monitor conversions between predefined steps. The most common visualization way of funnels is Sankey diagrams. Product managers or business analysts can create various funnels. For instance, for check-out process, we can examine some steps such as Add Product to The Basket => Go to Checkout Page => Choose Payment Option => Check Out. Then we can create some segments of people who bounce in this funnel and set a marketing trigger to send them an email like “Do you need help for check-out.”
However, in the real world, sometimes funnels may be more complicated than this. For example, customers may visit other products before the checkout. This may affect our funnel, and we may consider them as bouncers. For this kind of scenarios, we may define some of our steps as an “optional” step in our funnel. Also, a lot of analytics solutions allow defining time condition to steps for users to complete the goal within a specific timeframe from the previous or first goal.
Fallouts: Fall out analyses are the combination of the page flow analyses and the funnel analyses. Product managers and business analysts can see the detailed user behavior between funnel steps to understand why people are bouncing and where they are going. This kind of insight may helpful to create more accurate funnels or user flows.
Cohorts: Cohorts are well-designed visualization way of the measure the desired metric for conditioned target groups. Product managers should decide “Granularity” of the cohort with “Inclusion Metric” and “Returning Metric” before running the cohort. Inclusion metric helps you to define conditioned target group such as All Visitors or Buyers etc. And Returning Metric will help you to choose which metric you want to measure out of Inclusion group. The “Visit” can be an excellent example for returning metric.
The most common use of the cohorts is for monitoring the “Retention Rate” of visitors for specific pages or actions.
Flows: Flow analyses let users visualize a sequence of any dimension or dimension item. Also, flow analyses are good to visualize multi-dimensional sequences too. For instance, Entry flow of the “Campaigns” dimension. If you add “Pages” as a next dimension, you can easily visualize how many visitors are coming by campaigns and which pages they are landing. The most common visualization way of flow analyses is “Sankey Diagram” in the market.
Product managers can visualize “Entry Dimensions,” “Exit Dimensions” or just any “Dimension Item’s” flow in single or multiple dimension perspective.
Anomaly Analyses: The most common use of these analyses is visualizing the standard deviation levels for each data point and show anomalies (exceed or underperform) by supporting area chart with a line chart.
Product managers can notice anomalies in their main metrics and drill down to sub-metrics to understand the reason for the anomaly. But nowadays, some of the enterprise analytics solutions are providing also related sub-KPIs of that anomaly.
Analysis Variant Comparison: Analysis variant comparison let users compare the metric with different times ranges or segments or both at the same time.
As I mentioned, I will be deep diving into analyses and corporate metrics in my next blog posts, but I’d like to mention Dave McClure‘s “Startup Metrics for Pirates”. It may give a basic overview of the main metrics that product managers and marketeers should monitor and deep dive.
According to Dave McClure‘s “Startup Metrics for Pirates” figure above; category represents the maturity of the customer regarding customer journey. User status shows the metric(s) that product managers can monitor for that step. Conversion represents avg. conversion rate for that funnel step and estimated value doesn’t represent the cost of the user; it represents the avg. value of each user for that step.
Product managers should use right analysis to reach useful insights for data-driven decision makings. But also, the term of “digital analytics” is way more than visualizing the data. I will be deep diving into this topic in my future posts.
Customer feedbacks are strategic and crucial for giving customer-oriented product decisions. But gathering, organizing and evaluating customer feedbacks should be in a proper way to be able to consider useful customer feedbacks and avoid non-useful ones.
For gathering and organizing feedbacks, you should build topic-oriented breakdowns for feedback creation process and cluster those feedbacks around current product features or future concepts.
For instance, according to the figure above, we have three different scenarios for Tag Integration – Containers – Container Action – Delete function.
Scenario 1 is representing a bug, which should be clustered around of that feature and be solved with other clustered bugs. Amount and type of bugs can give a feeling about the maturity of the feature in the product.
Scenario 2 represents a problem of costumer which looks like a bug. But in this scenario, let’s say that deleted containers are disappearing in 24 hrs. because of some cache issues. In this case, the product should warn the customer about this condition. This is why we can consider this scenario as a knowledge center enhancement or UX improvement.
Scenario 3 represents a feature request of a customer for that specific function. These kinds of requests should be label as a feature request to consider in next version/iterations for this feature. If the business value prioritizes Tag Integration improvements, this feedback should be considered for go/not to go decision with other requests.