gmailctl
This utility helps you generate and maintain Gmail filters in a declarative way.It has a Jsonnet configuration file that aims to besimpler to write and maintain than using the Gmail web interface, to categorize,label, archive and manage your inbox automatically.
Table of contents
- gmailctl
Motivation
If you use Gmail and have to maintain (like me) a lot of filters (to applylabels, get rid of spam or categorize your emails), then you probably have (likeme) a very long list of messy filters. At a certain point one of your messagesgot mislabled and you try to understand why. You scroll through that horriblemess of filters, you wish you could find-and-replace stuff, test the changes onyour filters before applying them, refactor some filters together... in a waytreat them like you treat your code!
Gmail allows one to import and export filters in XML format. This can be used tomaintain them in some better way... but dear Lord, no! Not by hand! That's whatmost other tools do: providing some kind of DSL that generate XML filters thatcan be imported in your settings... by hand [this is the approach of the popularantifuchs/gmail-britta forexample].
Gmail happens to have also a neat API that we can use to automate the importstep as well, so to eliminate all manual, slow tasks to be done with the Gmailsettings.
This project then exists to provide to your Gmail filters:
- Maintainability;
- An easy to understand, declarative, composable language;
- A builtin query simplifier, to keep the size of your filters down (Gmail hasa limit of 1500 chars per filter);
- Ability to review your changes before applying them;
- Automatic update of the settings (no manual import) in seconds.
Install
gmailctl is written in Go and requires at least Go version 1.15. Make sure tosetup your $GOPATH correctly,including the bin subdirectory in your $PATH.
go get -u github.com/mbrt/gmailctl/cmd/gmailctl
Alternatively, if you're on macOS, you can install easily via Homebrew:
brew install gmailctl
You can also choose to install the snap:
sudo snap install gmailctl
If so, make sure to configure xdg-mime to open the config file with your favoriteeditor. For example, if you'd like to use vim:
xdg-mime default vim.desktop text/x-csrc
Once installed, run the init process:
gmailctl init
This will guide you through setting up the Gmail APIs and update yoursettings without leaving your command line.
Usage
The easiest way to use gmailctl is to run gmailctl edit. This will open thelocal .gmailctl/config.jsonnet file in your editor. After you exit the editorthe configuration is applied to Gmail. See Configuration forthe configuration file format. This is the preferred way if you want to startyour filters from scratch.
NOTE: It's recommended to backup your current configuration before you applythe generated one for the first time. Your current filters will be wiped andreplaced with the ones specified in the config file. The diff you'll get duringthe first run will probably be pretty big, but from that point on, all changesshould generate a small and simple to review diff.
Migrate from another solution
If you want to preserve your current filters and migrate to a more saneconfiguration gradually, you can try to use the download command. This willlook up at your currently configured filters in Gmail and try to create aconfiguration file matching the current state.
NOTE: This functionality is experimental. It's recommended to download thefilters and check that they correspond to the remote ones before making anychanges, to avoid surprises. Also note that the configuration file will be quiteugly, as expressions won't be reconstructed properly, but it should serve as astarting point if you are migrating from other systems.
Example of usage:
# download the filters to the default configuration file
gmailctl download > ~/.gmailctl/config.jsonnet
# check that the diff is empty and no errors are present
gmailctl diff
# happy editing!
gmailctl edit
Often you'll see imported filters with the isEscaped: true marker. This tellsgmailctl to not escape or quote the expression, as it might contain operatorsthat have to be interpreted as-is by Gmail. This happens when the downloadcommand was unable to map the filter to native gmailctl expressions. It'srecommended to manually port the filter to regular gmailctl operators beforedoing any changes, to avoid unexpected results. Example of such conversion:
{
from: "{foo bar baz}",
isEscaped: true,
}
Can be translated into:
{
or: [
{from: "foo"},
{from: "bar"},
{from: "baz"},
],
}
Other commands
All the available commands (you can also check with gmailctl help):
apply Apply a configuration file to Gmail settings
debug Shows an annotated version of the configuration
diff Shows a diff between the local configuaration and Gmail settings
download Download filters from Gmail to a local config file
edit Edit the configuration and apply it to Gmail
export Export filters into the Gmail XML format
help Help about any command
init Initialize the Gmail configuration
test Execute config tests
Configuration
NOTE: The configuration format is still in alpha and might change in thefuture. If you are looking for the deprecated versions v1alpha1, orv1alpha2, please refer to docs/v1alpha1.md anddocs/v1alpha2.md.
The configuration file is written in Jsonnet, that is a very powerfulconfiguration language, derived from JSON. It adds functionality such ascomments, variables, references, arithmetic and logic operations, functions,conditionals, importing other files, parameterizations and so on. For moredetails on the language, please refer to the officialtutorial.
Simple example:
// Local variables help reuse config fragments
local me = {
or: [
{ to: 'pippo@gmail.com' },
{ to: 'pippo@hotmail.com' },
],
};
// The exported configuration starts here
{
version: 'v1alpha3',
// Optional author information (used in exports).
author: {
name: 'Pippo Pluto',
email: 'pippo@gmail.com'
},
rules: [
{
filter: {
and: [
{ list: 'geeks@newsletter.com' },
{ not: me }, // Reference to the local variable 'me'
],
},
actions: {
archive: true,
labels: ['news'],
},
},
],
}
The Jsonnet configuration file contains mandatory version information, optionalauthor metadata and a list of rules. Rules specify a filter expression and a setof actions that will be applied if the filter matches.
Filter operators are prefix of the operands they apply to. In the example above,the filter applies to emails that come from the mail list 'geeks@newsletter.com'AND the recipient is not 'me' (which can be 'pippo@gmail.com' OR'pippo@hotmail.com').
We will see all the features of the configuration file in the followingsections.
Search operators
Search operators are the same as the ones you find in the Gmail filterinterface:
from: the mail comes from the given addressto: the mail is delivered to the given addresssubject: the subject contains the given wordshas: the mail contains the given words
In addition to those visible in the Gmail interface, you can specify nativelythe following common operators:
list: the mail is directed to the given mail listcc: the mail has the given address as CC destinationbcc: the mail has the given address as BCC destinationreplyto: the mail has the given address as Reply-To destination
One more special function is given if you need to use less common operators1, or want to compose your query manually:
query: passes the given contents verbatim to the Gmail filter, withoutescaping or interpreting the contents in any way.
Example:
{
version: 'v1alpha3',
rules: [
{
filter: { subject: 'important mail' },
actions: {
markImportant: true,
},
},
{
filter: {
query: 'dinner AROUND 5 friday has:spreadsheet',
},
actions: {
delete: true,
},
},
],
}
Logic operators
Filters can contain only one expression. If you want to combine multiple of themin the same rule, you have to use logic operators (and, or, not). Theseoperators do what you expect:
and: is true only if all the sub-expressions are also trueor: is true if one or more sub-expressions are truenot: is true if the sub-expression is false.
Example:
{
version: 'v1alpha3',
rules: [
{
filter: {
or: [
{ from: 'foo' },
{
and: [
{ list: 'bar' },
{ not: { to: 'baz' } },
],
},
],
},
actions: {
markImportant: true,
},
},
],
}
This composite filter marks the incoming mail as important if:
- the message comes from "foo", or
- it is coming from the mailing list "bar" and not directed to "baz"
Reusing filters
Filters can be named and referenced in other filters. This allows reusingconcepts and so avoid repetition. Note that this is not a gmailctl functionalitybut comes directly from the fact that we rely on Jsonnet.
Example:
local toMe = {
or: [
{ to: 'myself@gmail.com' },
{ to: 'myself@yahoo.com' },
],
};
local notToMe = { not: toMe };
{
version: 'v1alpha3',
rules: [
{
filter: {
and: [
{ from: 'foobar' },
notToMe,
],
},
actions: {
delete: true,
},
},
{
filter: toMe,
actions: {
labels: ['directed'],
},
},
],
}
In this example, two named filters are defined. The toMe filter gives a nameto emails directed to 'myself@gmail.com' or to 'myself@yahoo.com'. The notToMefilter negates the toMe filter, with a not operator. Similarly, the tworules reference the two named filters above. The name reference is basicallycopying the definition of the filter in place.
The example is effectively equivalent to this one:
{
version: 'v1alpha3',
rules: [
{
filter: {
and: [
{ from: 'foobar' },
{
not: {
or: [
{ to: 'myself@gmail.com' },
{ to: 'myself@yahoo.com' },
],
},
},
],
},
actions: {
delete: true,
},
},
{
filter: {
or: [
{ to: 'myself@gmail.com' },
{ to: 'myself@yahoo.com' },
],
},
actions: {
labels: ['directed'],
},
},
],
}
Actions
Every rule is a composition of a filter and a set of actions. Those actions willbe applied to all the incoming emails that pass the rule's filter. These actionsare the same as the ones in the Gmail interface:
archive: true: the message will skip the inbox;delete: true: the message will go directly to the trash can;markRead: true: the message will be mark as read automatically;star: true: star the message;markSpam: false: do never mark these messages as spam. Note that setting thisfield totrueis not supported by Gmail (I don't know why);markImportant: true: always mark the message as important, overriding Gmailheuristics;markImportant: false: do never mark the message as important, overridingGmail heuristics;category: <CATEGORY>: force the message into a specific category (supportedcategories are "personal", "social", "updates", "forums", "promotions");labels: [list, of, labels]: an array of labels to apply to the message. Notethat these labels have to be already present in your settings (they won't becreated automatically), and you can specify multiple labels (normally Gmailallows only one label per filter).forward: 'forward@to.com': forward the message to another email address. Theforwarding address must be already in your settings (Forwarding and POP/IMAP >Add a forwarding address). Gmail allows no more than 20 forwarding filters.Only one address can be specified for one filter.
Example:
{
version: 'v1alpha3',
rules: [
{
filter: { from: 'love@gmail.com' },
actions: {
markImportant: true,
category: 'personal',
labels: ['family', 'P1'],
},
},
],
}
Labels
You can optionally manage your labels with gmailctl. The config contains alabels section. Adding labels in there will opt you in to full labelmanagement as well. If you prefer to manage your labels through the GMail webinterface, you can by all means still do so by simply omitting the labelssection from the config.
Example:
{
version: 'v1alpha3',
// optional
labels: [
{ name: 'family' },
{ name: 'friends' },
],
rules: [
{
filter: { from: 'love@gmail.com' },
actions: {
labels: ['family'],
},
},
],
}
To make this work, your credentials need to contain permissions for labelsmanagement as well. If you configured gmailctl before this functionality wasavailable, you probably need to update your 'Scopes for Google API' in the'OAuth content screen' by adding https://www.googleapis.com/auth/gmail.labels.If you don't know how to do this, just reset and re-create your credentialsfollowing the steps in:
$ gmailctl init --reset
$ gmailctl init
If you want to update your existing config to include your existing labels, thebest way to get started is to use the download command and copy paste thelabels field into your config:
$ gmailctl download > /tmp/cfg.jsonnet
$ gmailctl edit
After the import, verify that your current config does not contain unwantedchanges with gmailctl diff.
Managing the color of a label is optional. If you specify it, it will beenforced; if you don't, the existing color will be left intact. This is usefulto people who want to keep setting the colors with the Gmail UI. You can findthe list of supported colorshere.
Example:
{
version: 'v1alpha3',
labels: [
{
name: 'family',
color: {
background: "#fad165",
text: "#000000",
},
},
],
rules: [ // ...
],
}
Note that renaming labels is not supported because there's no way to tell thedifference between a rename and a deletion. This distinction is importantbecause deleting a label and creating it with a new name would remove it fromall the messages. This is a surprising behavior for some users, so it'scurrently gated by a confirmation prompt (for the edit command), or by the--remove-labels flag (for the apply command). If you want to rename a label,please do so through the GMail interface and then change your gmailctl config.
Tests
You can optionally add unit tests to your configuration. The tests will beexecuted before applying any changes to the upstream Gmail filters or by runningthe dedicated test subcommand. Tests results can be ignored by passing the--yolo command line option.
Tests can be added by using the tests field of the main configuration object:
{
version: 'v1alpha3',
rules: [ /* ... */ ],
tests: [
// you tests here.
],
}
A test object looks like this:
{
// Reported when the test fails.
name: "the name of the test",
// A list of messages to test against.
messages: [
{ /* message object */ },
// ... more messages
],
// The actions that should be applied to the messages, according the config.
actions: {
// Same as the Actions object in the filters.
},
}
A message object is similar to a filter, but it doesn't allow arbitraryexpressions, uses arrays of strings for certain fields (e.g. the to field),and has some additional fields (like body) to represent an email as faithfullyas possible. This is the list of fields:
from: <string>: the sender of the email.to: [<list>]: a list of recipients of the email.cc: [<list>]: a list of emails in cc.bcc: [<list>]: a list of emails in bcc.replyto: <string>: the email listed in the Reply-To field.lists: [<list>]: a list of mailing lists.subject: <string>: the subject of the email.body: <string>: the body of the email.
All the fields are optional. Remember that each message object represent oneemail and that the messages field of a test is an array of messages. A commonmistake is to provide an array of messages thinking that they are only one.Example:
{
// ...
tests: [
messages: [
{ from: "foobar" },
{ to: ["me"] },
],
actions: {
// ...
},
],
}
This doesn't represent one message from "foobar" to "me", but two messages, onefrom "foobar" and the other to "me". The correct representation for that wouldbe instead:
{
// ...
tests: [
messages: [
{
from: "foobar",
to: "me",
},
],
actions: {
// ...
},
],
}
NOTE: Not all filters are supported in tests. Arbitrary query expressionsand filters with isEscaped: true are ignored by the tests. Warnings aregenerated when this happens. Keep in mind that in that case your tests mightyield incorrect results.
Tips and tricks
Chain filtering
Gmail filters are all applied to a mail, if they match, in a non-specifiedorder. So having some if-else alternative is pretty hard to encode by hand. Forexample sometimes you get interesting stuff from a mail list, but also a lot ofgarbage too. So, to put some emails with certain contents in one label and therest somewhere else, you'd have to make multiple filters. Gmail filters howeverlack if-else constructs, so a way to simulate that is to declare a sequence offilters, where each one negates the previous alternatives.
For example you want to:
- mark the email as important if directed to you;
- or if it's coming from a list of favourite addresses, label as interesting;
- of if it's directed to a certain alias, archive it.
Luckily you don't have to do that by hand, thanks to the utility library comingwith gmailctl. There's a chainFilters function that does exactly that: takesa list of rules and chains them together, so if the first matches, the othersare not applied, otherwise the second is checked, and so on...
// Import the standard library
local lib = import 'gmailctl.libsonnet';
local favourite = {
or: [
{ from: 'foo@bar.com' },
{ from: 'baz@bar.com' },
{ list: 'wow@list.com' },
],
};
{
version: 'v1alpha3',
rules: [
// ... Other filters applied in any order
]
// And a chain of filters
+ lib.chainFilters([
// All directed emails will be marked as important
{
filter: { to: 'myself@gmail.com' },
actions: { markImportant: true },
},
// Otherwise, if they come from interesting senders, apply a label
{
filter: favourite,
actions: { labels: ['interesting'] },
},
// Otherwise, if they are directed to my spam alias, archive
{
filter: { to: 'myself+spam@gmail.com' },
actions: { archive: true },
},
]),
}
To me
Gmail gives you the possibility to write literally to:me in a filter, to matchincoming emails where you are the recipient. This is going to mostly work asintended, except that it will also match emails directed to me@example.com.The risk you are getting an email where you are not one of the recipients, but ame@example.com is, is pretty low, but if you are paranoid you might considerusing your full email instead. The config is also easier to read in my opinion.You can also save some typing by introducing a local variable like this:
// Local variable, referenced in all your config.
local me = 'myemail@gmail.com';
{
version: 'v1alpha3',
rules: [
{
// Save typing here.
filter: { to: me },
actions: {
markImportant: true,
},
},
],
}
Directly to me
If you need to match emails that are to you directly, (i.e. you are not in CC,or BCC, but only in the TO field), then the default Gmail filter to: mymail@gmail.com is not what you are looking for. This filter in fact(surprisingly) matches all the recipient fields (TO, CC, BCC). To make this workthe intended way we have to pull out this trick:
local directlyTo(recipient) = {
and: [
{ to: recipient },
{ not: { cc: recipient } },
{ not: { bcc: recipient } },
],
};
So, from all emails where your mail is a recipient, we remove the ones whereyour mail is in the CC field.
This trick is conveniently provided by the gmailctl library, so you can use itfor example in this way:
// Import the standard library
local lib = import 'gmailctl.libsonnet';
local me = 'pippo@gmail.com';
{
version: 'v1alpha3',
rules: [
{
filter: lib.directlyTo(me),
actions: { markImportant: true },
},
],
}
Automatic labels
If you opted in for labels management, you will find yourself often having toboth add a filter and a label to your config. To alleviate this problem, you canuse the utility function lib.rulesLabels provided with the gmailctl standardlibrary. With that you can avoid providing the labels referenced by filters.They will be automatically added to the list of labels.
Example:
local lib = import 'gmailctl.libsonnet';
local rules = [
{
filter: { to: 'myself@gmail.com' },
actions: { labels: ['directed'] },
},
{
filter: { from: 'foobar' },
actions: { labels: ['lists/foobar'] },
},
{
filter: { list: 'baz' },
actions: { labels: ['lists/baz', 'wow'] },
},
];
// the config
{
version: 'v1alpha3',
rules: rules,
labels: lib.rulesLabels(rules) + [{ name: l } for l in [
'manual-label1',
'priority',
'priority/p1',
]],
}
The resulting list of labels will be:
labels: [
// Automatically added
{ name: 'directed' },
{ name: 'lists' }, // Implied parent label
{ name: 'lists/baz' },
{ name: 'lists/foobar' },
{ name: 'wow' },
// Manually added
{ name: 'manual-label1' },
{ name: 'priority' },
{ name: 'priority/p1' },
]
Note that there's no need to specify the label lists, because even if it's notused in any filter, it's the parent of a label that is used.
Things to keep in mind / gotchas:
- Removing the last filter referencing a label will delete the label.
- The only thing managed by the function is the list of labels names. You needto apply some transformations yourself if you want other properties (e.g. thecolor).
- If you have labels that are not referenced by any filters (maybe archivelabels, or labels applied manually). You have to remember to specify themmanually in the list.
Thanks to legeana for the idea!
Multiple Gmail accounts
If you need to manage two or more accounts, it's useful to setup bash aliasesthis way:
alias gmailctlu1='gmailctl --config=$HOME/.gmailctlu1'
alias gmailctlu2='gmailctl --config=$HOME/.gmailctlu2'
You will then be able to configure both accounts separately by using one orthe other alias.
Known issues
- Apply filters to existing emails: gmailctl doesn't support thisfunctionality for security reasons. The project currently needs only verybasic permissisons, and applying filters to existing emails requires fullGmail access. Bugs in gmailctl or in your configuration won't screw up yourold emails in any way, so this is an important safety feature. If you reallywant to do this, you can manually export your rules with
gmailctl export > filters.xml, upload them by using the Gmail Settings UIand select the "apply new filters to existing email" checkbox.
Comparison with existing projects
gmail-britta has similarmotivations and is quite popular. The difference between that project andthis one are:
gmail-brittauses a custom DSL (versus Jsonnet ingmailctl)gmail-brittais imperative because it allows you to write arbitrary Rubycode in your filters (versus pure declarative forgmailctl)gmail-brittaallows one to write complex chains of filters, but they feelvery hardcoded and fails to provide easy ways to write reasonably easy filters2.gmail-brittaexports only to the Gmail XML format. You have to import thefilters yourself by using the Gmail web interface, manually delete the filtersyou updated and import only the new ones. This process becomes tedious veryquickly and you will resort to quickly avoid using the tool when in a hurry.gmailctlprovides you this possibility, but also allows you to review yourchanges and update the filters by using the Gmail APIs, without you having todo anything manually.gmailctltries to workaround certain limitations in Gmail (like applyingmultiple labels with the same filter) and provide a generic query language toGmail,gmail-brittafocuses on writing chain filtering and archiving in veryfew lines.
In short gmailctl takes the declarative approach to Gmail filtersconfiguration, hoping it stays simpler to read and maintain, doesn't attempt tosimplify complex scenarios with shortcuts (again, hoping the configurationbecomes more readable) and provides automatic and fast updates to the filtersthat will save you time while you are iterating through new versions of yourfilters.
Footnotes
1: See Search operators you can use withGmail
2:
Try to write the equivalent of this filter with gmail-britta:
local spam = {
or: [
{ from: 'pippo@gmail.com' },
{ from: 'pippo@hotmail.com' },
{ subject: 'buy this' },
{ subject: 'buy that' },
],
};
{
version: 'v1alpha3',
rules: [
{
filter: spam,
actions: { delete: true },
},
],
}
It becomes something like this:
#!/usr/bin/env ruby
# NOTE: This file requires the latest master (30/07/2018) of gmail-britta.
# The Ruby repos are not up to date
require 'rubygems'
require 'gmail-britta'
SPAM_EMAILS = %w{foo@gmail.com bar@hotmail.com}
SPAM_SUBJECTS = ['"buy this"', '"buy my awesome product"']
puts(GmailBritta.filterset(:me => MY_EMAILS) do
# Spam
filter {
has [{:or => "from:(#{SPAM_EMAILS.join("|")})"}]
delete_it
}
filter {
has [{:or => "subject:(#{SPAM_SUBJECTS.join("|")})"}]
delete_it
}
end.generate)
Not the most readable configuration I would say. Note: You also have to makesure to quote the terms correctly when they contain spaces.
So what about nesting expressions?
local me = 'pippo@gmail.com';
local spam = {
or: [
{ from: 'foo@gmail.com' },
{ from: 'bar@hotmail.com' },
{ subject: 'buy this' },
{ subject: 'buy that' },
],
};
{
version: 'v1alpha3',
rules: [
{
filter: {
and: [
{ to: me },
{ from: 'friend@mail.com' },
{ not: spam },
],
},
actions: { delete: true },
},
],
}
The reality is that you have to manually build the Gmail expressions yourself.