Martin Joo

« back published by @mmartin_joo on May 25, 2022

How To Use Laravel Pipelines To Implement More Advanced Filters

This article is a short case study where I show you how you can create more complicated filters using Laravel pipelines, the strategy design pattern, and enums. This example comes from an e-mail marketing app or e-mail services provider, such as MailChimp or ConvertKit. It describes how to apply filters when sending a broadcast, so first, let's discuss what a broadcast is. It's a one-time e-mail for subscribers.

It has only a handful of important properties:

  • Subject: this is what you see in your favorite e-mail client.
  • Content: this is the actual HTML content of the e-mail. It contains the link where you downloaded this chapter.
  • Filters: you can apply filters so only a subset of subscribers will receive the e-mail. For example, you don't want to send a promotion e-mail to people who already bought the product.

In this article, I'll discuss how to implement such a feature in a cool and easy-to-extend way.

Data Model

In this example every subscriber has a form ID and some tags. A form is literally an HTML form where you can subscribe to a newsletter, and subscribers can also have tags. We want to apply filters based on these two things. Here are the table structure:

Subscriber Data Model

As you can see:

  • A subscriber belongs to one form.
  • And belongs to many tags.

The main components we're going to use:

  • A Pipeline that takes care of filtering subscribers.
  • An Enum that can be used as a factory.
  • Some polymorphic classes that implement the strategy design pattern.

To send broadcasts, first, we need to filter the subscribers based on the filters in the broadcast. Let's define the relationship between filters. If you have the following setup:

  • Form IDs: 1
  • Tag IDs: 10,12

then the following query will run:

where
    subscribers.form_id IN (1)
and
    subscriber_tag.tag_id IN (10,12)

Or in other words:

  • There's an AND relation between the form and the tag filters
  • There's an OR relation between the IDs. Writing tag_id IN (1,2) is the same as writing tag_id = 1 OR tag_id = 2

Filters are a topic that can require many changes in the future. Just think about some of these features:

  • What if the app has some e-commerce features? For example, creators can publish products on the platform, and subscribers can buy them if they're interested. Now we have filters like: who purchased a product, who used a discount code, how much a subscriber paid for products, etc.
  • The app does not have this feature right now, but what about custom fields for subscribers? In this case, users can create any fields they want and populate via some automation rules, by hand, or by CSV import. All of these fields can be used in filters.
  • Geolocation based on IP addresses?
  • What if we need to integrate with 3rd parties? Maybe these integrations also bring some new interesting filter opportunities.

The Strategy

We clearly need a solution that can handle new use-cases easily. In my opinion, the strategy pattern is our best choice here. In this pattern, every filter has a dedicated class, and there's some kind of abstraction. Either an abstract class or an interface. As of now, we need three classes:

  • Filter: This is the abstract class or interface that defines the contract of a concrete filter class.
  • TagFilter: This class takes care of the tag filters.
  • FormFilter: And finally, a class for form filters.

These filters can be expressed as one query, so it seems logical to me that both of these classes will append where clauses to a base query. Something like that:

public function filterSubscribers(Broadcast $broadcast)
{
  // $query is a basic select * from subscribers
  $query = Subscriber::query();

  /*
   * At this point $query contains something like:
   * select * from subscribers
   * where subscribers.form_id IN (1)
   */
  (new FormFilter($broadcast->filters))->filter($query);  
  
  // Finally TagFilter will join the subscriber_tag table and add another where clause
  (new TagFilter($broadcast->filters))->filter($query);  
}

It's just pseudo-code, but let's think about the idea. Can this work in other use cases?

Custom subscriber fields

I think this can be implemented with two extra tables:

  • fields: it contains every user-defined field with types (such as string, number, JSON) and other meta information.
  • subscriber_field: a pivot table where every row represents the value of a custom field for a specific subscriber.

If you think about it, this is very similar to tags. It is also implemented with two tables: tags and subscriber_tag. So if we can handle tags with this pattern, I think custom fields can also be handled.

Geolocation

In this situation, we probably will have some new columns in the subscribers table, such as:

  • IP address
  • Country
  • Region
  • City

Or maybe a new table to handle multiple IP addresses, multiple locations. But the point is: it's pretty easy.

3rd party integrations

This is very hard to predict, but basically, we have two choices:

  • Calling the 3rd party API every time before sending a broadcast. This is clearly a very poor implementation. Imagine calling an API 5000 times before sending a broadcast to 5000 subscribers.
  • Storing the "external" data in our database. For example, integrating with an e-commerce provider and we want to store every product purchase made by our subscribers. In this case, it's just another table with a subscriber_id that can be joined and filtered with a where clause.

E-commerce features

It's also tough to predict, but if we're talking about products and purchases, it's essentially the same as storing the purchases given by a 3rd party API.

In my opinion, all of those cases can be implemented by adding a new Filter class such as the FormFilter or the TagFilter. The main difference will be the data. In the case of forms or tags, we'll only use IDs, but in other cases (such as custom fields or geolocation), a more sophisticated data structure will be needed. But you can always invent some more generic DTO class to handle such challenges.

Now, back to reality! Let's start with the FormFilter, the easiest of all:

namespace Domain\Subscriber\Filters;

use Illuminate\Database\Eloquent\Builder;

class FormFilter
{
  public function __construct(protected readonly array $ids)
  {
  }
  
  public function filter(Builder $subscribers): Builder
  {
    if (count($this->ids) === 0) {
      return $subscribers;
    }

    return $subscribers->whereIn('form_id', $this->ids);
  }
}

Two important things:

  • There's no abstraction yet. We'll get there later. At this point, I have no idea what's the abstraction is, so I like to start with one of the concrete implementations.
  • This class lives inside the Subscriber domain. I know you're reading the "Broadcasts" chapter, but these filter classes will filter subscribers.

The filter method is pretty simple, but there are two critical things. First of all, the if statement is very important. Without it, the following query will run:

select * from subscribers
where form_id in ()

Since it's an empty array, every subscriber will be returned. It's clearly not what we want. If the $ids array is empty, we don't need to do anything. There's no form filter; there's nothing to handle.

Now, let's see the TagFilter:

namespace Domain\Subscriber\Filters;

use Illuminate\Database\Eloquent\Builder;

class TagFilter
{
  public function __construct(protected readonly array $ids)
  {
  }

  public function filter(Builder $subscribers): Builder
  {
    if (count($this->ids) === 0) {
      return $subscribers;
    }

    return $subscribers->whereHas('tags', fn (Builder $tags) =>
      $tags->whereIn('id', $this->ids)
    );
  }
}

It queries subscribers where they have tags, but only those tags specified in the $ids array. Now we can see the abstraction:

  • A method that takes a Builder and returns another one.
  • A __construct with an array of integers.

It's time to write the Filter base class:

namespace Domain\Subscriber\Filters;

use Domain\Mail\DataTransferObjects\FilterData;
use Illuminate\Database\Eloquent\Builder;

abstract class Filter
{
  public function __construct(protected readonly array $ids)
  {
  }

  abstract public function filter(Builder $subscribers): Builder;
}

The constructors are removed from the filter classes, and we have a contract. When I use the strategy pattern and have multiple classes that implement the same contract, I usually want some "dynamic" behavior. Let me illustrate it with some made-up code:

public function filterSubscribers(Broadcast $broadcast)
{
  /*
   * $broadcast->filters is a FilterData that can be converted to an array: 
   * [
   *    'tag_ids' => [1,2],
   *    'form_ids' => [1],
   * ];
   */
  
  $query = Subscriber::query();
  
  foreach ($broadcast->filters->toArray() as $type => $ids)
  {
    // $type is 'tag_ids' and $ids is [1,2]
    $filter = $this->createFilter($type, $ids);
    $query = $filter->filter($query);
  }
}

So by "dynamic," I mean I don't want to use TagFilter or FormFilter manually. I want something that loops through the broadcast's filters and gets things done. One of the key components of this "something" is the $this->createFilter() method. This is a factory method. It creates the appropriate Filter instance from the tag_ids or form_ids strings. Luckily it can be achieved with an Enum:

namespace Domain\Subscriber\Enums;

use Domain\Subscriber\Filters\Filter;
use Domain\Subscriber\Filters\FormFilter;
use Domain\Subscriber\Filters\TagFilter;

enum Filters: string
{
  case Tags = 'tag_ids';
  case Forms = 'form_ids';

  public function createFilter(array $ids): Filter
  {
    return match ($this) {
      self::Tags => new TagFilter($ids),
      self::Forms => new FormFilter($ids),
    };
  }
}

This single enum comes with several benefits:

  • It's a single source of truth for every possible filter.
  • It helps us be consistent with the string tag_ids and form_ids.
  • It can be used as a factory that creates a Filter instance from an enum value.

The createFilter can be used as:

$filter = Filters::from('tag_ids')->createFilter($ids);

The from function creates an enum instance from a scalar value (only works with backed enums), and we can call methods on this instance.

Pipelines

The whole process of filtering can be imagined as a "pipeline" where we send the initial query through the Filter instances, and each filter modifies the query. Maybe you don't know about this, but Laravel has a concept exactly like that:

use Illuminate\Pipeline\Pipeline;

$subscribers = app(Pipeline::class)
  ->send(Subscriber::query())
  ->through($arrayOfFilters)
  ->thenReturn()
  ->get();

Let's see what's happening here step by step:

  • An instance of Pipeline is resolved from the container.
  • The initial query (select * from subscribers) is passed to the send method.
  • We can specify an array of "pipes" in the through method. This array will contain the actual Filter instances.
  • The thenReturn will return the result after each "pipe" is executed. It's the final query with the appropriate where clauses.
  • Since it's a Builder instance, we need to call the get method to get the actual subscribers.

So Laravel will send the initial query through the Filter instances where both TagFilter and FormFilter will append a where clause (and a join) to the query builder.

To use the Pipeline, we have to make a small change in the Filter classes:

class TagFilter extends Filter
{
  public function handle(Builder $subscribers, Closure $next): Builder
  {
    if (count($this->ids) === 0) {
      return $next($subscribers);
    }

    $subscribers->whereHas('tags', fn (Builder $tags) =>
        $tags->whereIn('id', $this->ids)
    );

    return $next($subscribers);
  }
}

There are three differences compared to the previous version:

  • The function is called handle. This is because Pipeline will look for a handle method in each pipe. It can be overwritten, but I decided to go with the default.
  • There's a second argument. It's called next and is a type of Closure. Each pipe will get the next pipe as an argument. So when the first Filter is being executed, let's say the FormFilter, the $next argument will be the second filter, so the TagFilter instance.
  • We call this $next variable with the query builder instance on the last line.

Of course, we also need to change the TagFilter:

class FormFilter extends Filter
{
  public function handle(Builder $subscribers, Closure $next): Builder
  {
    if (count($this->ids) === 0) {
      return $next($subscribers);
    }

    $subscribers->whereIn('form_id', $this->ids);

    return $next($subscribers);
  }
}

And the abstract Filter as well:

abstract class Filter
{
  public function __construct(protected readonly array $ids)
  {
  }

  abstract public function handle(Builder $subscribers, Closure $next): Builder;
}

You absolutely don't need to use this approach, but it fits the current use case very well. After all of this, let's write a class that will actually use these classes and returns the subscribers for a broadcast:

namespace Domain\Subscriber\Actions;

class FilterSubscribersAction
{
  /**
   * @return Collection<Subscriber>
   */
  public static function execute(Broadcast $broadcast): Collection
  {
    return app(Pipeline::class)
      ->send(Subscriber::query())
      ->through(self::filters($broadcast))
      ->thenReturn()
      ->get();
  }

  /**
   * @return array<Filter>
   */
  public static function filters(Broadcast $broadcast): array
  {
    return collect($broadcast->filters->toArray())
      ->map(fn (array $ids, string $key) => 
        Filters::from($key)->createFilter($ids)
      )
      ->values()
      ->all();
  }
}

Let's focus on the filters method first. $broadcast->filters returns a FilterData that can be converted into an array by calling the toArray. It has the following shape:

[
  'tag_ids' => [1,2],
  'form_ids' => [3,4],
]

So in the map $key is either tag_ids or form_ids and $ids contain the actual IDs. As we discussed earlier, this line would create a TagFilter:

Filters::from('tag_ids')->createFilter([1,2]);

Since the initial collection is associative, we need to call the values method to get a classic array with integer indexes. This array can be used with the Pipeline.

Here's a context diagram to wrap things up:

Filtering Subscribers

This whole example comes from my 259-page book Domain-Driven Design with Laravel. If you liked it, check out the landing page for more details. You can download another 49-page sample chapter that describes the basic concepts of domain-driven design:

Domain-Driven Design with Laravel