An open response

George Entenman recently wrote a blog post where he expressed confusion over some of the things I talk about, build, or endorse. I wanted to take a moment to respond as I’m sure he’s not alone in this.

George writes:

I’m getting confused by all this Brett Terpstra stuff.

It’s ok, I’m constantly confused. It’s what drives me to look for and create solutions.

I’m getting confused because I can’t tell what all these apps are for or how exactly they differ. I don’t want to use more apps than necessary.

As I say often, I’m a fan of “the right tool for the job,” even if that means using more tools than I’d hoped would be necessary. That’s not to say that I don’t appreciate a good multi-function tool, but you’ll find that most of the tools I espouse do One Thing Well¹. My only requirements for any tool are that its data be compatible with my other tools (usually plain text) and that I can search it (usually with Spotlight) along with everything else.

In fact, by the time this post is done I’ll probably have introduced a dozen more apps to the mix. They work together for me, personally, but I point them out not with the expectation that you’ll adopt every one, but to provide options for honing your own workflow and organization setup.

To directly answer the statement, though, I accept that I tend to gloss over the use cases when presenting new tools. I (usually errantly) expect everyone to just see the practical uses and possibilities without my having to detail them. That’s not always fair, and I accept it as constructive criticism even if it wasn’t meant as criticism at all.

I want to search through my documents the way nvALT does. But I don’t want to use a monolithic database to store them.

Most worthwhile search tool are going to create a database index of your files and their content. You can (and I often do) use tools like grep, ack and spot on the command line to find what you’re looking for. Something like Spotlight, though, functions as that monolithic database while still keeping all of your text free to roam around.

Almost all of the tools I use store plain text files or have the ability to export them easily. Apps like Found and Spotlight make it possible to search quickly (relatively, in Spotlight’s case) through all of these files. Where Spotlight fails, tools like nvALT provide a way to search a specific type of information. I usually know what “bucket” to look in, and can pick the best search tool for that type of data.

It doesn’t make sense to me to store, for example, Day One type journal entries in with all of my random notes. I prefer Day One’s calendar approach and nvALT’s note-taking approach to one giant collection of text. I use them differently. This doesn’t mean I don’t want to be able to search them all together.

I don’t want to dictate how anyone else should use all of these tools, and I’ve seen plenty of setups that vastly differ from mine in their usage of said tools, but the individual workflows function just fine (if not better than my own). I’ve seen nvALT become a CMS/blogging engine. I’ve seen Day One actually used as a life journal (weird, right?). I’ve seen people use complex file-naming patterns instead of content search. They’re all valid; I’m not ever going to suggest that my personal uses for these tools is the “right” way to use them. I leave that up to you guys to decide.

I want to attach some metadata to a markdown file. Why can’t my markdown entries have tags, especially for searching or creating blogs? If I could specify a filename, then something like nvALT could store entries in files without spaces in their names while allowing more verbose subjects.

There are a few options for doing this, and you have to pick what’s going to work best for you.

First, you can use MultiMarkdown metadata at the top of text files. This isn’t indexed by any tool I know of, but it’s easily searchable in tools like nvALT, Spotlight and from the command line.

Next, you can use OpenMeta tags. I’ve long used Tags.app to quickly apply tags to my files, photos and documents. These are searchable in Spotlight, just like the rest of my system.

Lastly, file-naming conventions are great, especially if you have a little scripting capability. Turning 2012-08-22_my_response_to_george.blog.md into a blog post with a headline of “My Response To George” and a metadata line with the date is one line of code. I know that not everyone has the tools to do that, but coming up with a good convention and sticking to it can provide options even if you have to seek out other tools to make use of them.

I want to convert my markdown files to HTML and other formats. I can do this with Marked, for example, but I wish I could get it to save the output automatically. Can it have a CLI?

First, automation is coming to Marked, eventually. In the meantime, yes, there are plenty of CLIs available for doing conversions. MultiMarkdown can create your HTML file (CSS and JavaScript can be specified in metadata), and comes with scripts for converting to a variety of other formats (ODF, OPML, PDF).

There are a dozen other command-line utilities for converting Markdown as well. Discount is my favorite after MultiMarkdown. Then there’s kramdown and maruku, Python Markdown 2, the original Perl version from John Gruber, PHP Markdown and more. Each has its own syntax, and I try to avoid using many of the extensions in my daily note-taking in order to keep my notes cross-compatible with all of them. This basically means sticking with the original Markdown syntax.

I want standardized markdown. Of course, I can see ways to improve the standard and understand why so many apps make little improvements. I would enhance the standard myself if I could: why, for example, do I have to put my own numbers in to create a numbered list?

I covered my thoughts on standardized Markdown above, but wanted to point out that you can do the following and still get a numeric list:

1. Item 1
* Item 2
* Item 3

As long as the first item at a given nest level in the list is a digit followed by a period, it will turn into:

1. Item 1
2. Item 2
3. Item 3

I want to view my markdown files in a calendar. I love Day One’s calendar view - this is an idea that I’ve wanted for a long time.

Personally I would love an app that could take a number of files and display them in a calendar view. It should be part of the conversion software mentioned above.

The problem with this is that you need structured data to be able to accomplish this. This is the reason that Day One uses XML files. XML is plain text with added markup. It’s just as searchable and relatively easy to extract data from. For an app like Day One to function quickly and efficiently, though, you need an easily-parseable way to determine things like dates, attachments, etc.

I realize that this comes back to the question of multiple file formats. There are sometimes reasons for varying formats, though, and Day One is a perfect example. It can export all of my entries as plain text, though, and it’s easy to add notes from plain text files using its CLI, so you could easily have a system that syncs certain notes from nvALT (using MMD metadata or a file-naming convention) to Day One, and export your Day One entries to nvALT. A lot of the things I automatically send to Day One also get logged in nvALT so that I have more options for consuming it later.

OPML is another structured data markup that makes a great transport system and allows for additional metadata. Apps like MultiMarkdown Composer and a variety of outlining and mind mapping apps across platforms can work with it, and most of them can return plain text. It’s a transport system, just like Day One’s XML. My ultimate landing spot is plain text, but when the need arises for structured data, XML and OPML are (to me) perfectly valid ways to make plain text into something easier to work with programatically.

I want collapsable, rearrangeable outlining. Even better is collapsable outlining where you can rearrange the outlines by moving the collapsed sections.

Many of the best Markdown editors out there (MultiMarkdown Composer, Byword and friends) as well as Marked provide headline navigation. Collapsing is harder to find. You might be in luck, though: FoldingText from Hog Bay Software is turning into an amazing editor for Markdown and plain text files, and allows quick collapsing and expansion of sections based on headers. It’s in beta and not feature-complete yet, but it’s already a winner. Have a look.

I’ll be writing about some of FoldingText’s features when it’s closer to release, but if you grab it be sure to look at the Expert Users Guide under help. It will overwhelm you, I guarantee it. It’s one of those cases, though, where you can use it as a simple Markdown editor and never even know that all of those advanced options exist. Very elegant, but I digress.

If you use something like FoldingText (or MultiMarkdown Composer/Byword) as an external editor for nvALT notes stored as plain text, you have a great editing system and the speedy Notational Velocity search.

I’d like standardized, open file formats. The software that Brett likes uses multiple formats.

There’s markdown ASCII format. There’s Day One’s XML and nvALT’s binary database.

Some programs use the file system; others use a database.

Since I’ve already addressed this (I think), I’ll point out some specifics:

1. nvALT has an option to store notes as individual text files. Anyone working with the systems I create should be using that. The bonus there is that nvALT still stores its database to keep search speedy, but you never notice that there are two storage systems running. You just know that you get lightning fast search and still have access to all of your notes as plain text files.
2. Day One’s XML is very simple, and it’s barely more than a text file. They’re stored on disk and readily accessible, just like nvALT notes.
3. Ultimately, everything comes down to plain text for me. Day One exports it, nvALT bathes in it, my text and Markdown editors eat it all day long, and Spotlight loves it. I can manipulate it and display it in myriad ways, and search it with ease.

So here’s the summary: I create a lot of tools, and I share a lot of links to other people’s work. I don’t expect anyone to use them all; even I don’t. I try to fill gaps where I see them and fix problems I spot. I do use a lot of tools overall, and I imagine my workflows would baffle anyone not intimately familiar with all of them. They work seamlessly for me, though; if they don’t, I leave them behind.

Moving forward I’ll try to do a better job of explaining use cases when presenting new tools so it takes less work to decide whether they’re worth including in your workflow. Hopefully that will make what I share more useful in the end.