Payment routing part #2: Developing Paddle's rules engine

A deep dive into Paddle's rules engine that determines payment routing destinations, featuring fallback and weighted strategies, data-driven improvements, and future machine learning plans.

Over the past year and a half, we’ve built substantially more routing and retry logic to Paddle Billing, added a bunch of new payment methods, and opened several new merchant accounts to take advantage of local acquiring.

To make this possible, we first needed to standardize payments from different payment service providers (PSPs), which we explored in the first blog post.

Then we built out a rules engine to determine how payments are routed to their destinations, maximizing outcomes for our users. In this blog post, we’ll dive into how the rules engine works.

Rules engine for payment routing

Every time a payment is attempted on our platform, information about the payment is sent to the rules engine which decides which merchant account and which PSP integration should handle the payment.

Our payments experts define and continually optimize active rules in the rules engine based on our data and analytics. They do this using a bespoke app we built that lets them update the order in which rules should be evaluated in a simple drag-and-drop interface.

It might be considered a luxury to have a UI to manage the rules, instead of treating them as business logic that is part of our payment solution. However, treating the rules as data rather than code allows payment routing logic to be managed independently of our engineering team.

---
title: Payment routing and retries
---
graph TD
    A[Payment request] --> C[Rules engine 🔀]
    B[Rules UI] --> C
    C --> D[Matched rule]

    subgraph Strategy ["Strategy & Routing"]
        E[Fallback strategy]
        F[Split test strategy]
        G[Destinations]
    end

    D --> Strategy

    D --> H[Outcomes from attempts]
    H --> I[Analytics pipeline]
    I --> J[Data warehouse & visualization]

    G --> K[Authorization attempt]
    K --> L[Response mapping]
    L --> M[Retry logic]
    M --> G

    classDef process fill:#f9f9f9,stroke:#333,stroke-width:2px
    classDef engine fill:#e8f5e8,stroke:#4caf50,stroke-width:2px
    classDef strategy fill:#333,color:#fff,stroke:#333,stroke-width:2px
    classDef destinations fill:#ffab91,stroke:#ff5722,stroke-width:2px
    classDef analytics fill:#bbdefb,stroke:#2196f3,stroke-width:2px
    classDef authorization fill:#fff3e0,stroke:#ff9800,stroke-width:2px

    class A,B,D,H process
    class C engine
    class E,F strategy
    class G destinations
    class I,J analytics
    class K,L,M authorization

To explain how the rules engine works, let’s follow a hypothetical payment through the system step-by-step. The first step is to load the latest version of all our rules that were set using the UI.

Each rule has a defined set of conditions which are basic logical statements that apply to a set of dimensions:

• Dimensions include things like payment method, payment amount, currency, card brand, card issuer, customer country, and payment type (one-off or subscription).
• Conditions include things like Payment amount > $10", and Card issuer = Chase Bank USA.

If a transaction is a renewal payment for a subscription, we might also take into account which destination was used on previous attempts for the same specific payment method.

Rules are evaluated in order until a rule is found where all the conditions are met. We call this rule the matched rule. The matched rule provides the destination which represents the PSP and merchant account.

The most basic rule might have a single destination, but rules can also have one of the following strategies that define multiple destinations, how those destinations will be attempted, and in what order.

Fallback strategy

Rules have fallback strategies. This lets us set alternative destinations if the first destination could not accept the payment.

In the previous blog post, we talked about response mapping. One part of that process is deciding whether the failure reason is deemed suitable for retrying at a different destination. For example, if there are system reliability issues or a kind of “soft decline” from the issuing bank, we might be safe to retry.

There’s no limit to how many fallbacks can be defined in a rule’s strategy, but in practice we tend not to use more than two.

Weighted strategy

Sometimes we want to to distribute payments matching a rule across multiple destinations. This is particularly useful if we are slowly rolling out a new merchant account. The main benefit is being able to benchmark different destinations against each other to identify which destination drives better results under which conditions.

For example, we might want to determine which PSP performs better for Norwegian credit cards purchasing subscriptions.

The next step for our hypothetical payment is to execute the strategy, choosing which destination we will use first.

Once we’ve contacted the PSP and mapped the response, we have the option of trying again if a payment doesn’t succeed. Depending on the specifics of the PSP, we may retry the payment at the same destination but with slightly different parameters. These tend to be small optimizations that we’ve developed in partnership with our payment providers to help achieve better acceptance rates at the margin.

With the weighted strategy, the first step is to see which destination has been specified. Then, we attempt the payment against that selected merchant account. If that payment fails but it can be retried and if the rule has a fallback strategy, then we’ll try again with the merchant account set in the fallback strategy.

Once all retries are exhausted, we have our final result of the transaction attempt. All outcomes of all attempts are saved for analysis which we use to improve our routing rules or retry behavior.

Data-driven improvements to payment routing

It’s not much use developing a way to route payments if you don’t know what routing rules to deploy!

We record the outcome of every payment in a unified manner for all payment methods and PSP integrations. This makes sure we have a standardized understanding of what happened. We typically store the exact failure reasons and decline codes from the integration, and also our internal unified representation of the outcome.

We also store which payment method, PSP, merchant account, and Paddle legal entity was used, along with an identifier representing the matched rule. This is crucial. By storing a reference of the version of the rule, we can account for any changes made to the rule over time.

This data is logged for every transaction, regardless of outcome. It’s then replicated and made available in a data warehouse using a data pipeline developed by our data engineering team.

Once prepared, our product analysts use this information to conclude outcomes from split tests, produce new rules, and develop new hypotheses for testing.

Even if a conclusion on an A/B test is reached with a high degree of confidence, we always maintain a small portion of payment attempts to the losing variant as a control group. This is because our payment partners actively evolve their integrations and risk rules over time. As we continually test, we’re able to effectively identify when we should reconsider our own rules.

Challenges with multiple payment destinations

It’s worth acknowledging some of the downsides of taking on the complexities of having multiple payment destinations.

On the technical side, there are a lot of workflows to support alongside payments. Every integration needs to support fraud detection mechanisms, refund procedures, and different ways of managing disputes. This makes every PSP integration a considerable engineering effort, since we have to build and maintain the routing system, plus all of the other workflows.

And it’s not just technical considerations. Coordinating payments in different regions means setting up individual legal entities and corresponding bank accounts in regions across the world.

Every region has different regulatory requirements, and each provider has a slightly different procedure for settling funds. Our legal and financial teams are involved every step of the way when rolling out a new payment destination and bear the cost of managing a more complex treasury.

Looking to the future

With the technical investments we’ve made, we have a solid foundation to grow. There are still a few regions that represent a promising market for digital products that we’ve not yet expanded into. Our next step is to increase the number of merchant accounts held around the world, building out the legal and financial structures required.

We’re also planning to increase our analytical modelling capabilities with some machine-learning techniques. Our approach up until recently has been to run relatively straightforward small-scale A/B tests on simple hypotheses, and then to adjust the rules accordingly. Developing more sophisticated models that use large amounts of historical test data will let us deploy much more complex rules that consider many more dimensions than we have today.

It’s an exciting prospect with some interesting engineering challenges in store and we’ll be sure to share our progress along the way! With a unified payment lifecycle, we’re able to standardize payments from different PSPs, measure performance, and build a rules engine that determines how each payment gets routed.