Different types of product manager: A worked example

In my last post I talked about 4 dimensions that determine what kind of product manager an organization needs:

• Number of employees in the organization
• Number of users for your product
• Nature of the product
• Culture of the organization

Different types of organization and product expect/require different types of product management. We all say “product management” but we can mean very different things when we say it, depending on where the organization and product is along these four dimensions.

Here is an example based on a real experience of mine to illustrate this. It was when I was at Rated People, probably about 2010. It shows how the decision-making around the product (i.e. the product management) was informed by these four dimensions.

Rated People is a marketplace where home-owners post jobs they need done. Each job is notified to relevant tradespeople who can then choose to bid for that job or not. The system figures out if a tradesperson is relevant based on the type of work they do (plumbing, carpentry etc) and which areas they work in.

Which areas they work in was originally set by giving a central address and a distance. The system would calculate a circle using the distance as the radius. This was problematic because it didn’t reflect the reality of where people would travel to for work. Famously, London is split into North and South parts by the river Thames. Crossing the river can be time-consuming. People on the north side are more likely to spend more time on the north side of the river and those on the south to stay on the south side more.

We had three different ideas about how to address this problem.

First approach: Pave The Cow Paths. User testing showed that tradespeople generally thought of their work areas based on postcodes. So we could have come up with a solution that would let you enter a list of postcodes somehow, either by clicking on a map or by writing them into a form.

Second approach: Use the latest tech. In 2010 we now had the ability to draw any shape on a map. This seemed quite neat but when we then put it through user testing, our users found it very frustrating to draw the shape they wanted. This feature would have been a complete non-starter for our users.

Third approach: Lateral-thinking application of new tech. This is what we ended up going with. We approximated a circle with an octagon and allowed the user to drag the corners around. This worked well because if you were happy with the auto-generated circular area, the octagon was pretty much good enough for you. If you wanted to change the area then you could pretty well draw any shape you wanted by dragging the corner points. Hat tip to the mighty Oleg Roshka who did all the heavy lifting to make this feature happen.

The Lateral-thinking application of new tech suited us at the time because of where we were against the four dimensions:

• We were a relatively small organization and so could easily get a few key people on a call together to figure out the right solution together
• We had a relatively high number of customers (10’s of thousands) who weren’t particularly tech-savvy
• We wanted this part of the product to be a differentiator compared to the competition (the tradespeople are the paying side of the marketplace)
• The founder liked relatively elegant tech solutions that generalized well

It’s not a case that the other approaches were worse. They would have been more appropriate in different circumstances. The use the latest tech could have worked if our user base was more tech-savvy, for example. Pave the cow paths could have worked but it would have needed a fair amount of fine-tuning after release because different people, in different parts of the country, needed different levels of granularity to define their work areas.

The organization valued and needed this Lateral-thinking application of new tech at the time. That doesn’t suit all products and all organizations.

Here’s an example where pave the cow paths was the right approach. It’s from no less than Facebook and their photos product in 2005:

[E]very development in the Facebook photos product can be linked to that moment in 2005 when the team saw how people were using the photos: “Instead of trying to change people’s behaviour, they codified it.”

https://www.mindtheproduct.com/paving-cow-path-and-other-stories-by-simon-cross/

Facebook started 2005 with 1 million active users and ended with 5 million. This would have meant a lot of engineering challenges around scalability and stability and/or a lot of firefighting to keep the site up and working well. Accel Partners made their $13m investment into Facebook in 2005 which would have meant a lot of hiring. Based on this, some educated guessing about how Facebook would have measured up against the four dimensions:

1. Relatively small organization but going through a lot of hiring and organizational change
2. Very large user base who were young and relatively tech-savvy
3. The product team was up against more established competitors in the photo space, and so had to find a relatively simpler way of getting more activity on Facebook photos
4. The culture was about “moving fast and breaking things”

In this context, it made a lot of sense for the user-facing side of the PM team to get to grips with user behaviour and codify it. If it didn’t work to increase user activity they could pivot to a different idea.

I’m using Facebook as an example because it’s a well-known company that was explicitly following a pave the cow paths philosophy. Please don’t fall victim to survivorship bias and assume that dropping a 2005 Facebook PM into another organization would magically fix things¹.

It’s horses for courses. Different organizations and products expect and need different things from their product managers². We make things harder for ourselves by describing this all under one job title: “product manager” when in reality we mean a lot of different things depending on the context.

With all this said, do you want to see the Rated People feature I was talking about? Of course you do, and you’re in luck because it seems this feature worked well from a user point of view: it’s still there nearly 15 years later.

Here’s an example from DJC Contractors. I’m going to imagine that they live somewhere around the L or the o of London given that they are happy to go further north-west than they are south-east.

Here’s one for Comforooms ltd who are keen on the Western side of London.

On the other hand, Light Electrics, below, works on the Eastern side of London and surrounding areas. They are happy to work in Dartford but apparently don’t want to go to Croydon or as far as Watford. So I’m going to guess they are based relatively close to Chelmsford.

Or getting away from the capital, below are Next Gen Driveways who cover the whole of the South West of England, Swansea, Cardiff and Newport but have explicitly excluded some major local cities/towns (Bristol and Bath).

1. If anything, this Facebook example is a good counterpoint to the prevailing wisdom that you should not be paving cow paths, depending on the context of the organization and the product ↩︎
2. Even in this Facebook example, I’d expect that a PM working on the advertiser side of the business would treat things differently because those are the paying customers. ↩︎

How to figure out what type of product manager an organization needs

The (software) product manager job role is far from standardized. It can mean very different things in different organizations. Check how your organization maps across four dimensions to figure out what kind of product manager it needs. And to determine if it’s the right place for you as a PM.

I used to think of product management as a linear progression. Starting from the very first days of a startup, where a founder is driving the whole product vision, to a professionalised, product-led approach.

Something like the pioneer/settler/town planner model, (with some further useful discussion on it here)

Or along the lines of this model from Gartner:

To be fair, the Pioneers/Settlers/Town Planners and the Gartner models aren’t necessarily straightforward maturity models. But they do a good job of articulating different approaches to product management.

What they don’t tell you is where on the model you should be.

I used to think it was just down to the scale of your organization. A smaller one would be more pioneering or instinct-driven, while a larger one would need to become more product-led, or risk becoming a dead-end feature factory.

It turns out that it’s more nuanced than that¹. Here are four dimensions that will help clarify what an organization needs when they say product manager:

• Number of employees in the organization
• Number of users
• Nature of the product
• Culture of the organization

There will often be correlation between these, but not always.

Number of employees in the organization

As the organization gets larger, Metcalfe’s law becomes more of a challenge: The more people you have, the number of potential touchpoints goes up exponentially.

A smaller organization is more about individuals. The stakeholder management aspect product role is more about your relationships with specific (probably highly-opinionated) individuals.

As the organization scales the stakeholder management piece becomes much more about aligning multiple stakeholders, handling cross-functional teams and removing organizational obstacles.

Number of users

A product with 10 users is going to feel very different to one with 100 which, in turn will feel very different to one with 1000 or 10,000 etc.

At 10 users, the person running product can intimately know what everyone needs, perhaps better than the users themselves. At 100, one person can still have an overview of all the users but you can start thinking about ideal customer profiles and all that good stuff. If you’re in the 10’s of thousands then user research, customer testing and user segmentation take on a whole new meaning.

Nature of the product

Let’s simplify for a moment and call the (software) product we’re creating a website. Even in this simplified scenario it’s clear that not all websites are created equal.

Assume you’re an airline. You need a website so that people can find your flights, book them, get their loyalty points, get customer support, handle changes and other problems, etc. But the website is not your product. Your product is flying people from one place to another.

Similar story if you’re a hotel or a retailer where the software product is an enabler, it’s not your real product.

Very different story if you’re selling a SaaS product. In this case your web app is your product. Similar if you’re a marketplace: for sure you need the buyers and the sellers on there but your web app is far more central to your product than in the airline or hotel or retailer example.

If you are selling your web app then the nature of the product role is more technical than if you’re not selling your web app. By “more technical” I mean that understanding the art of the possible is a much more important part of the product role than in a similar scale organization where the website because you are likely to be trying to differentiate your product from your competitors more.

Culture of the organization

Imagine an organization that wants to move fast and break things.

Now imagine an organization working in a highly-regulated environment that has a much longer time-horizon.

These will need to take a very different approach to their product management. The move fast and break things culture will prioritise getting things out to users fast. The organization with a longer time-horizon will prioritize more planning, iteration, experimentation and validation before something goes live.

This is not to say that the organization with the longer time-horizon is necessarily going to be more sleepy than the “move fast and break things” one. You can still do experiments at pace even if you’re not delivering working products live. In my experience I’ve seen some products where the users value stability and so would prefer things to know about new features weeks ahead of time, and others where users would jump on a new thing if it can be released in the next few days.

Putting it all together

Here are a few examples based on places I’ve seen.

Company A: <100 employees (more personal than cross-team); >25,000 customers (lots of user testing, user research); B2B2C marketplace (relatively technical understanding needed); Engineering-heavy culture (needed to be relatively nerdy)

Company B: >5,000 employees (lots of cross-functional alignment needed); < 10 customers (new product, so needed to differentiate); B2B SaaS product (relatively technical understanding needed); culture valued slow and steady planning with a long time horizon

Company C: <100 employees; >100 customers; Services company – tech was mainly an internal tool (so didn’t need too much technical know-how); very commercial-focussed culture focussed on building features to help win new clients

Each of these companies had very different needs in their product managers. Not to say one is better and one is worse. Just that, for example, if a Company A or B PM found themselves in Company C they might feel like they were in a feature factory. Conversely, a Company A or C PM would feel stifled in Company B.

1. Credit to an inspiring conversation with Andy Richardson, among others, at a London CTOs unconference session about the pros and cons of the CPTO title, followed up by some discussions with the ever-challenging Chirag Shah. ↩︎

Clicking on Garbage Captcha

Feels like these days it would be easier for an AI to deal with a captcha than for a human being to complete it.

Today I had possibly my worst captcha-related experience.

This was not deemed acceptable.

I had to also click the middle one on the right. Even though it needs quite a lot of imagination to see it as a set of traffic lights.

Then I was allowed to proceed.

Feeling like a proper grumpy old man having to deal with this nonsense.

I can imagine an LLM could hallucinate that the middle right image is a traffic lights. But a human being would have to be pretty high to come to the same conclusion.

Hallucinations are how GenAI works

There’s been a lot of talk about how ChatGPT (and similar) hallucinate when they generate text. It’s an unfortunate choice of words to use because it sounds like some kind of side-effect. But in reality it’s central to the way that Generative AI works.

Do you know why they’re called hallucinations?

Before we had ChatGPT making text generation popular, creating images was the the big thing in Generative AI. When researchers looked into how these images were being generated they saw something that looked, well, like hallucinations.

Here’s an example from DeepDream.

And of course we had many, many cases where image generation tools would “hallucinate” extra fingers or lopsided ears etc.

When you’re talking about image generation, a word like “hallucinate” works quite well. If you see a generated image that looks like an acid trip or has wonky ears in or whatever then you tend to consider the whole image a hallucinated image. You treat the whole image as not real.

But when you’re seeing a ChatGPT output with 3 incorrect facts in, you don’t think of the whole text as a hallucination. You tend to consider those 3 incorrect facts as a hallucination, distinct from the rest of the text. Weird how people tend to treat pictures as “one thing” but text as “a bunch of different things stitched together that we can consider separately”.

There have been attempts to use a different word when talking about text generation: Confabulation. According to the National Institute of Health, confabulation is:

a neuropsychiatric disorder wherein a patient generates a false memory without the intention of deceit

https://www.ncbi.nlm.nih.gov/books/NBK536961/

But too late! “Hallucination” had already taken its grip on the public’s imagination.

When you look at a ChatGPT output you should expect that the whole thing is one confabulation. This confabulation is often useful but it’s still just made up text, however cleverly it is made up.

There are all kinds of guardrails that people have to build around the LLMs to help guard against these confabulations/hallucinations. But ultimately, when you’re generating images or text with GenAI, you are asking a machine to hallucinate / confabulate for you. Please remember that.

Bonfire of the Best Practices

As we get stuck into 2024 I am wondering how many of the product and engineering best practices that we’ve seen develop over the past 10-15 years will turn out to be ZIRPs (“zero interest rate phenomenon”).

Probably quite a few. Two reasons I can see:

First reason is pretty obvious. When money was cheap the objective was to raise as much investment as possible and use that money to build an aggressive roadmap fast. When time is of the essence and cost isn’t a big deal, the first reaction of most managers is going to be to hire people to do more stuff. Not surprising that many companies over-hired – e.g. see https://qz.com/from-overhiring-to-optionality-what-we-can-learn-from-1850267076 and https://stripe.com/gb/newsroom/news/ceo-patrick-collisons-email-to-stripe-employees

Burning loads of VC cash to hire loads of people to build more stuff brings its own set of challenges. The processes and best practices that evolve will work well with these sorts of challenges.  When there is less investment money floating around you need to be leaner and more intentional in where you spend your time. The processes that work best in a leaner environment won’t necessarily be the same as those that worked in the sorts of VC-backed outfits we’ve seen over the last decade.

It’s a common trope that you shouldn’t just pick a process that worked in a company whose brand you admire and expect it to work in your company. What I’m saying here is a bit stronger than that. I’m saying that any product or engineering advice coming out of the VC-backed cash-burning-machine era might need to be chucked in the bin in these leaner times.

It’s not just that a bigger company’s processes might not work for where your company is right now. The sorts of processes that worked in a ZIRP era business model might just not make sense in a leaner era.

Here comes the second reason that makes it even more important for you to question ZIRP-era process advice.

As a company gets bigger, the people who have the time and inclination to write about processes get further and further removed from the reality on the ground. Even in @simfoni it’s difficult for me to know what an individual product iteration really involves for the people doing the work. If you’re reading a blog post by a tech leader of a company with 200  or 1,000 engineers, how confident should you be that the blog reflects reality? Or does it reflect what the writer thinks is happening, or how they would like things to be happening?

A great example of this is the Spotify Model. The Spotify model is (was?) an organizational structure that inspired a lot of imitators until it turned out that even Spotify didn’t use the Spotify model: https://www.agility11.com/blog/2020/6/22/spotify-doesnt-use-the-spotify-model  It was as much a leadership aspiration as it was a statement of reality.

To sum up, I see 2 things combining here:

1. Processes developed in ZIRP-era VC-backed company might fundamentally not work in leaner times
2. The processes as described in blog posts and analyst reports maybe never even existed in real life

So a 2024 resolution for myself, and a call to arms for anyone reading is to “question all the blog posts”. Even more than you normally would.

Except for this one, of course. This one is bang on the money 😀

IP infringement in the world of GenAI

Compelling debate on what looks like IP infringement from Gary Marcus on his Substack. It shows how an apparently trivial prompt can make MidJourney create pictures that look startlingly similar to stills from films.

Assume, for sake of the argument, that MidJourney was trained on these specific films, and that this counts as IP infringement. Seem like reasonable assumptions to me, but obviously I am not a lawyer.

Anyway, assuming the above, then who is liable?

[a] The people who train the model on content from films

[b] The person who inputs the prompt and then uses the resulting image as their own

There is a well-trodden debate about whether sites like Facebook or X are liable for the content on them. The consensus seems to be that the company is not liable for what people write on it.

Could that argument apply here, such that MidJourney is not held responsible for what human beings do with the stuff they create with it?

This is a thorny issue – a European Commission article from Feb 2023 has this to say:

the question of ownership and authorship of AI-generated works is not fully settled by the law yet, and as a “hot topic” may evolve in the years to come depending on regulatory changes and on case law. 

https://intellectual-property-helpdesk.ec.europa.eu/news-events/news/intellectual-property-chatgpt-2023-02-20_en

In my view there is a qualitative difference between running a GenAI model and hosting a social network. In a social network or other website, it is really easy to differentiate between the “code” (that the company writes) and the “data” (which comes from the users). With GenAI models, the data is interwoven into the model itself. Whoever trained the AI model consciously decided what content to train it on.

My first instinct, then, is that the model developers should be responsible for any IP infringement because the model developers are the ones who chose how to incorporate data into the model. My second instinct is to wonder whether taking this position cause more problems than it solves:

[a] in a world where people use open source models, and potentially train new models on top of open source models, does it make sense to hold the original model creator accountable?

[b] If you’ve got a chain of model developers, each one taking an existing model and training it further, then how could you prove where the IP infringement took place? It is conceivable that the original source data for a model is not even accessible in any useful way any more.

[c] If you end up saying that the model developer is infringing IP when its model outputs something that looks similar to someone else’s IP, then does this have impacts about the IP ownership of anything else generated by the model? If a model developer can breach copyright by plagiarising something, then does it own the copyright to anything it creates that is not plagiarism?

Where is the GenAI disruption going to happen?

I recently heard Saad Ansari (former Director of AI at Jasper) speaking about how he sees forthcoming evolution in the GenAI space. It was really thought provoking so here are some notes of mine from the talk.

He sees four key use cases for GenAI:

1. Co-piloting. Github users are already familiar with an AI tool called co-pilot. More generally, you can think of ChatGPT or similar as a co-pilot who is there to help you do your tasks, whether that is by drafting an email or a job spec for you or helping you learn a new topic of prepare for a meeting.
2. Personalization
3. Bringing everyone the “power of Pixar”
4. Robotics (both virtual agents and physical robots)

I’m going to go into a bit more on the personalization piece.

Go back far enough and the internet was all about search¹. You go to Google and get “about 8,400,000,000 results (0.35 seconds)”. Then you scan page 1 and possibly 2 to see if there’s anything relevant.

Then, over time, things become more personalised for the user. One high profile example was the Netflix Prize from 2009. This was a competition with a $1m prize to use machine learning to improve Netflix’s recommendation algorithm (“if you liked show X then probably you will like shows Y and Z”). At the time this ML work was pretty groundbreaking.

Now with GenAI we are in a new world again. In this world new things can be created to the user’s taste. Saad used the words “synthesis” and “remixing” to describe this. The GenAI models have seen enormous amounts of text, images, audio etc in their training which they can use to synthesise new things. They are like a music producer doing a remix. From their training data they can make something that is just what the user is interested in, has never fully existed before, but is similar to what it has been trained on.

What does this sea change in personalization mean for future disruption?

From this perspective, Saad believes, someone like Adobe or TurboTax is safe. It’s easier for them to enhance their products with GenAI than it is for a new GenAI entrant to add the core features that companies like this have.

On the other hand, someone like Amazon might not be safe. A more personalized shopping service could well disrupt them. Imagine a service like:

1. You upload some photos of your family
2. Based on the photos an AI figures out your interests
3. It gives you some ideas of local activities to do nearby
4. And gives you some links to things you might want to buy

Be honest, it sounds pretty realistic, doesn’t it?

Notes

1. Or you could go back a bit further to the dark days of domain dipping but it’s the same principle ↩︎

Styling with django-allauth

I implemented django-allauth into Syracuse recently.

I wanted to implement some simple styling. It was surprisingly tricky to piece together the relevant information from different stackoverflow and medium posts. So here is what I ended up with in case it’s useful for others.

The app source code including allauth is in the allauth_v01 tag. The relevant commit that included allauth is here.

Briefly the necessary changes are:

settings.py

1. Install allauth per the quickstart docs. I didn’t add the SOCIALACCOUNT_PROVIDERS piece as those can be set up via the admin interface as shown in this tutorial.
2. The DIRS section tells Django where to look locally for templates. If it doesn’t find templates in here then it will look in other magical places which would include the templates that come in the allauth library

Then it’s just a case of find which template you need from the allauth library. In my case I just wanted to add some basic formatting from the other pages in the app to the allauth pages. So I just had to copy https://github.com/pennersr/django-allauth/blob/main/allauth/templates/allauth/layouts/base.html to an equivalent location within my local templates directory and add in one line at the top:

{% include 'layouts/main-styling.html' %}

Simple when you know how, right.

Up until this point the styling file was living in the relevant app directory in the project, so I moved it to a shared location in templates where it can be accessed by any part of the project. The rest of the updates in the commit are

• related to moving this file and changing the rest of Syracuse to use the new file location.
• implementing a little snippet to give an appropriate message/links if users are logged in or not

More software development lessons from my side project

Last month I wrote about migrating the syracuse codebase to Neo4j and changing the hosting from Heroku to Digital Ocean.

Since then I’ve finished adding the remaining types of content that I have to the UI, so you can now see information on corporate finance activities, senior appointment activities and location-related activities (e.g. adding a new site, exiting a territory). This is all part of building up a picture of how an organization evolves over time using information extracted from unstructured data sources.

I want to write about two things that came up while I was doing this which reminded me of why some things are good do and some aren’t!

CSV, CSV, CSV

The bad news was that adding the location and appointment activities into the UI showed that there were some inconsistencies in how the different types were represented in the data. The good news was that the inconsistencies weren’t too hard to fix. All the data was stored as RDF triples in json-ld format. This made it pretty trivial to regenerate. It would have been a lot harder to do it if the data had been stored in a structured database. Once you start getting data into a database, then even the smallest schema change can get very complicated to handle. So I’m glad I follow the advice of one of the smartest developers I ever worked with: Don’t assume you need a database, if a CSV file can handle your requirements then go with that.

In fact one feature I implemented does use a CSV file as it’s storage. Easier than using a database for now. My preferred approach for data handling is:

1. CSV
2. JSON
3. Database

Test, Test, Test

Adding these new data types into the graph made the already quite messy code even messier. It was ripe for a refactor. I then had to do a fair amount of work to get things to work right which involved a lot of refreshing a web page, tweaking some code, then repeating.

My initial instinct was to think that I didn’t have time to write any tests.

But guess what… it was only when I finally wrote some tests that I got to the bottom of the problems and fixed them all. All it took was 2 integration tests and I quickly fixed the issues. You don’t need 100% code coverage to make testing worthwhile. Even the process of putting together some test data for these integration tests helped to identify where some of the bugs were.

The app is currently live at https://syracuse.1145.am, I hope you enjoy it. The web app is running on a Digital Ocean droplet and the backend database is in Neo4j’s Auradb free tier. So there is a fair amount of traffic going backwards and forwards which means the app isn’t super fast. But hopefully it gives a flavor.

Syracuse update: Django and Neo4j

Time for another update in my side project, following on from https://alanbuxton.wordpress.com/2023/08/05/rip-it-up-and-start-again-without-ripping-it-up-and-starting-again/

Since that post I’ve implemented a new backend with Neo4j and created an app for accessing the data¹. It’s here: https://github.com/alanbuxton/syracuse-neo. The early commits have decent messages showing my baby steps in getting from one capability to the next².

The previous app stored each topic collection separately in a Postgres database. A topic collection is the stories taken from one article, and there could be several stories within one article. This was ok as a starting point, but the point of this project is to connect the dots between different entities based on NLPing articles, so really I needed a Graph database to plug things together.

Heroku doesn’t support Neo4j so I’ve moved the app to a Digital Ocean VM that uses Neo4j’s free tier AuraDB. It’s hosted here http://syracuse.1145.am and has just enough data in it to fit into the AuraDB limits.

The graph visualization is done with vis.js. This turned out to be pretty straightforward to code: you just need some javascript for the nodes and some javascript for the edges. So long as your ids are all unique everything seems pretty straightforward.

Visualizing this sort of data in a graph makes it a lot more immediate than before. I just want to share a few entertaining images to show one feature I worked on.

The underlying data has a node for every time an entity (e.g. organization) is mentioned. This is intentional because when processing an article you can’t tell whether the name of a company in one article is the same company as a similarly-named company in a different article³. So each node in the graph is a mention of an organization and then there is some separate logic to figure out whether two nodes are the same organization or not. For example, if it’s a similar name and industry then it’s likely that the two are the same organziation.

This sometimes led to a ball of lines that looks like Pig-Pen’s hair.

On the plus side, this does make for a soothing visual as the graph library tries to move the nodes about into a sensible shape. With a bit of ambient sounds this could make a good relaxation video.

But, pretty though this may be, it’s hard to read. So I implemented an ‘uber node’ that is the result of clubbing together all the “same as” nodes. A lot more readable, see below:

Below is an example of the same graph after all the Accel Partners nodes had been combined together.

Next steps:

1. Implement the other types of topic collections into this graph (e.g. people appointments, opening new locations)
2. Implement a feature to easily flag any incorrect relationships or entities (which can then feed back into the ML training)

Thanks for reading!

Notes

1. With thanks to the https://github.com/neo4j-examples/paradise-papers-django for their example app which gave a starting point for working with graph data in Neo4j. ↩︎
2. For example, git checkout b2791fdb439c18026585bced51091f6c6dcd4f72 is a good one for complete newbies to see some basic interactions between Django and Neo4j. ↩︎
3. Also this type of reconciliation is a difficult problem – I have some experience of it from this project: https://theybuyforyou.eu/business-cases/ – so it’s safer to process the articles and then have a separate process for combining the topics together. ↩︎