iCalendar Support Part 4: preparing a patch!

Parsing real world data

Once I had the parser compliant with RFC5545, I also needed to make it handle data from the real world. To do this, I tested the parser against all the real iCalendar data I could get my hands on: email attachments from my own email archive.

Since I have a local copy of all my mail and I use Notmuch to index it, I wrote a quick Python script to grab all these attachments and save them as individual files:

import os, hashlib, notmuch

def main():
    ics_dir = "/some/output/directory/"
    ics_files = []
    db = notmuch.Database()
    query = db.create_query("tag:attachment and ('.ics' or 'text/calendar')")
    for mail in query.search_messages():
        for part in mail.get_message_parts():
            ct = part.get_content_type()
            fn = part.get_filename()
            if (ct == 'text/calendar' or str(fn).endswith(".ics")):
                bytes = part.get_payload(decode=True)
                file = hashlib.sha1(bytes).hexdigest() + ".ics"
                with open(ics_dir + file, "w+b") as sink:
                    sink.write(bytes)

if __name__ == '__main__':
    main()

I then ran the parser over all these files to find out where it would have problems.

I found that the most common error was improper escaping in text values. Unescaped commas in addresses were particularly prominent. (Commas need to be escaped in text because they are sometimes used as separators in lists of text values.) So I added a special case to the parser to loosen up text parsing when unescaped text creates no ambiguities.

Other than that, my personal data showed that different implementations make different mistakes; there were no errors that were so common that they were worth adding further exceptions to the parser. Instead, I added some hooks to the parser so users can write their own functions to clean up bad data before it’s parsed, and an option to parse strictly according to 5545, mostly for debugging purposes.

Dealing with real world data also requires signaling errors in the right place in the parser, and handling them appropriately. I added a simple error handling framework to icalendar.el based on compilation-mode, and added error handling to the parser so that parsing can continue at the next place that makes sense. A bad parameter value won't prevent the parser from parsing the rest of a property; a bad property won't prevent parsing the rest of a component; and so on.

All of this should make it pretty easy for Emacs users to make use of iCalendar data that isn't perfectly RFC-compliant. It will also, I hope, provide enough quality-of-life improvements for developers that they can improve both Emacs' and other implementations. Combined with the syntax highlighting in icalendar-mode (see Part 1), the debugging features I've added to the parser should make Emacs a great choice for anyone working on any iCalendar implementation.

Diary import and export

Once I had all that working, I was finally ready to tackle the job that icalendar.el does: converting between iCalendar data and Emacs diary entries. I decided to reimplement this from the ground up, because in my opinion the code in icalendar.el is just too difficult to work with and not worth saving.

My main goal here was to make both import and export more flexible. I also fixed a number of inconsistencies and gotchas along the way, like the fact that there was previously no way to control the date format used for importing to the diary, even though the diary itself allows great flexibility in date formats.

I decided to switch to skeleton.el templates to make it easier for users to customize the import format. Now, instead of customizing a hierarchy of inflexible format strings, you can write an import template like this:

(define-skeleton my-event-skeleton
  "Just the date, time, summary, and location on one line"
  nil
  date & " " & start-to-end & " " & summary & (when location " @ ") & location
  "\n")
(setq diary-icalendar-vevent-skeleton-command #'my-event-skeleton)

which will import events to your diary like:

2025/11/2 10:30AM Brunch @ The Restaurant

Since a skeleton is just an Elisp function, you have all the power of Elisp available in such templates. You could e.g. dispatch to other skeletons depending on the event data, or apply filters to certain properties based on the sender.

I've added a lot more customization variables in the diary-icalendar group that provide control over import and export, and I've added support for iCalendar's VTODO and VJOURNAL components. I've also made it possible to #include an iCalendar file from your diary file, so that you can mark and display its events in the calendar on the fly, without having to import them to the diary first. This will be useful if e.g. you want to mainly keep your calendar data on a server somewhere, but download it to a local file periodically for display in Emacs.

I decided that idempotent export and reimport of diary entries was not one of my design goals. The diary format is far too flexible and underspecified to make this a reasonable goal. You can pretty much write diary entries however you like, in free form text, as long as there is some kind of date at the beginning. The diary also has no syntax for time zones. It's thus basically impossible to roundtrip text from diary to iCalendar without either imposing further constraints on the user or losing data.

That means one cannot expect to use the diary export and import to sync with an external calendar server. But being able to #include an iCalendar file in the diary ameliorates this: you don't have to sync between the formats; you can view the iCalendar data in the diary and calendar in Emacs without syncing.

Despite these changes, I was careful to prioritize backward compatibility. Users who are already happy with their icalendar.el setup can continue using it, and they should also be able to switch to the new implementation without changing their setup (just use the new diary-icalendar-* commands corresponding to the old icalendar-* commands, e.g. diary-icalendar-import-buffer). I also ported the existing icalendar.el tests to the new implementation, so the new implementation should handle all edge cases that icalendar.el already did, and a lot more.

What's next

I still have to rebase against master and prepare a patch, which hopefully I'll submit that to the original bug report sometime later today. I'm planning to submit the code for feedback from the maintainers, and work on updating the manual and the NEWS file while I'm waiting.