Use static strings for CustomError.

This resulted in a 20% speedup.

README.md

@@ -4,10 +4,31 @@ Organic is an emacs-less implementation of an [org-mode](https://orgmode.org/) p
 
 ## Project Status
 
-This project is a personal learning project to grow my experience in [rust](https://www.rust-lang.org/). It is under development and at this time I would not recommend anyone use this code. The goal is to turn this into a project others can use, at which point more information will appear in this README.
+This project is still under HEAVY development. While the version remains v0.1.x the API will be changing often. Once we hit v0.2.x we will start following semver.
+
+Currently, the parser is able to correctly identify the start/end bounds of all the org-mode objects and elements (except table.el tables, org-mode tables are supported) but many of the interior properties are not yet populated.
+
+### Project Goals
+- We aim to provide perfect parity with the emacs org-mode parser. In that regard, any document that parses differently between Emacs and Organic is considered a bug.
+- The parser should be fast. We're not doing anything special, but since this is written in Rust and natively compiled we should be able to beat the existing parsers.
+- The parser should have minimal dependencies. This should reduce effort w.r.t.: security audits, legal compliance, portability.
+- The parser should be usable everywhere. In the interest of getting org-mode used in as many places as possible, this parser should be usable by everyone everywhere. This means:
+  - It must have a permissive license for use in proprietary code bases.
+  - We will investigate compiling to WASM. This is an important goal of the project and will definitely happen, but only after the parser has a more stable API.
+  - We will investigate compiling to a C library for native linking to other code. This is more of a maybe-goal for the project.
+### Project Non-Goals
+- This project will not include an elisp engine since that would drastically increase the complexity of the code. Any features requiring an elisp engine will not be implemented (for example, Emacs supports embedded eval expressions in documents but this parser will never support that).
+- This project is exclusively an org-mode **parser**. This limits its scope to roughly the output of `(org-element-parse-buffer)`. It will not render org-mode documents in other formats like HTML or LaTeX.
+### Project Maybe-Goals
+- table.el support. Currently we support org-mode tables but org-mode also allows table.el tables. So far, their use in org-mode documents seems rather uncommon so this is a low-priority feature.
+- Document editing support. I do not anticipate any advanced editing features to make editing ergonomic, but it should be relatively easy to be able to parse an org-mode document and serialize it back into org-mode. This would enable cool features to be built on top of the library like auto-formatters. To accomplish this feature, We'd have to capture all of the various separators and whitespace that we are currently simply throwing away. This would add many additional fields to the parsed structs and it would add more noise to the parsers themselves, so I do not want to approach this feature until the parser is more complete since it would make modifications and refactoring more difficult.
+### Supported Versions
+This project targets the version of Emacs and Org-mode that are built into the [organic-test docker image](docker/organic_test/Dockerfile). This is newer than the version of Org-mode that shipped with Emacs 29.1. The parser itself does not depend on Emacs or Org-mode though, so this only matters for development purposes when running the automated tests that compare against upstream Org-mode.
 
 ## Using this library
-TODO: Add section on using Organic as a library (which is the intended use for this project).
+TODO: Add section on using Organic as a library (which is the intended use for this project). This will be added when we have a bit more API stability since currently the library is under heavy development.
+
+## Development
 
 ### The parse binary
 This program takes org-mode input either streamed in on stdin or as paths to files passed in as arguments. It then parses them using Organic and dumps the result to stdout. This program is intended solely as a development tool. Examples:

scripts/callgrind.bash

@@ -4,10 +4,23 @@ set -euo pipefail
 IFS=$'\n\t'
 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 
+: ${PROFILE:="perf"}
 
+function main {
+    local additional_flags=()
+    if [ "$PROFILE" = "dev" ] || [ "$PROFILE" = "debug" ]; then
+        PROFILE="debug"
+    else
+        additional_flags+=(--profile "$PROFILE")
+        # We have to disable avx512 because valgrind does not yet support it.
+        export RUSTFLAGS="-C target-feature=-avx512"
+    fi
 
-(cd "$DIR/../" && RUSTFLAGS="-C opt-level=0" cargo build --no-default-features)
-valgrind --tool=callgrind --callgrind-out-file="$DIR/../callgrind.out" "$DIR/../target/debug/parse" "${@}"
+    (cd "$DIR/../" && RUSTFLAGS="-C target-cpu=x86-64-v3" cargo build --no-default-features "${additional_flags[@]}")
+    valgrind --tool=callgrind --callgrind-out-file="$DIR/../callgrind.out" "$DIR/../target/${PROFILE}/parse" "${@}"
 
-echo "You probably want to run:"
-echo "callgrind_annotate --auto=yes '$DIR/../callgrind.out'"
+    echo "You probably want to run:"
+    echo "callgrind_annotate --auto=yes '$DIR/../callgrind.out'"
+}
+
+main "${@}"

src/compare/elisp_fact.rs

@@ -0,0 +1,474 @@
+use std::borrow::Cow;
+
+use crate::types::AngleLink;
+use crate::types::BabelCall;
+use crate::types::Bold;
+use crate::types::Citation;
+use crate::types::CitationReference;
+use crate::types::Clock;
+use crate::types::Code;
+use crate::types::Comment;
+use crate::types::CommentBlock;
+use crate::types::DiarySexp;
+use crate::types::Document;
+use crate::types::Drawer;
+use crate::types::DynamicBlock;
+use crate::types::Element;
+use crate::types::Entity;
+use crate::types::ExampleBlock;
+use crate::types::ExportBlock;
+use crate::types::ExportSnippet;
+use crate::types::FixedWidthArea;
+use crate::types::FootnoteDefinition;
+use crate::types::FootnoteReference;
+use crate::types::GreaterBlock;
+use crate::types::Heading;
+use crate::types::HorizontalRule;
+use crate::types::InlineBabelCall;
+use crate::types::InlineSourceBlock;
+use crate::types::Italic;
+use crate::types::Keyword;
+use crate::types::LatexEnvironment;
+use crate::types::LatexFragment;
+use crate::types::LineBreak;
+use crate::types::NodeProperty;
+use crate::types::Object;
+use crate::types::OrgMacro;
+use crate::types::Paragraph;
+use crate::types::PlainLink;
+use crate::types::PlainList;
+use crate::types::PlainListItem;
+use crate::types::PlainText;
+use crate::types::Planning;
+use crate::types::PropertyDrawer;
+use crate::types::RadioLink;
+use crate::types::RadioTarget;
+use crate::types::RegularLink;
+use crate::types::Section;
+use crate::types::SrcBlock;
+use crate::types::StatisticsCookie;
+use crate::types::StrikeThrough;
+use crate::types::Subscript;
+use crate::types::Superscript;
+use crate::types::Table;
+use crate::types::TableCell;
+use crate::types::TableRow;
+use crate::types::Target;
+use crate::types::Timestamp;
+use crate::types::Underline;
+use crate::types::Verbatim;
+use crate::types::VerseBlock;
+
+pub(crate) trait ElispFact<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str>;
+}
+
+pub(crate) trait GetElispFact<'s> {
+    fn get_elisp_fact(&'s self) -> &'s dyn ElispFact<'s>;
+}
+
+impl<'s, I: ElispFact<'s>> GetElispFact<'s> for I {
+    fn get_elisp_fact(&'s self) -> &'s dyn ElispFact<'s> {
+        self
+    }
+}
+
+impl<'s> GetElispFact<'s> for Element<'s> {
+    fn get_elisp_fact(&'s self) -> &'s dyn ElispFact<'s> {
+        match self {
+            Element::Paragraph(inner) => inner,
+            Element::PlainList(inner) => inner,
+            Element::GreaterBlock(inner) => inner,
+            Element::DynamicBlock(inner) => inner,
+            Element::FootnoteDefinition(inner) => inner,
+            Element::Comment(inner) => inner,
+            Element::Drawer(inner) => inner,
+            Element::PropertyDrawer(inner) => inner,
+            Element::Table(inner) => inner,
+            Element::VerseBlock(inner) => inner,
+            Element::CommentBlock(inner) => inner,
+            Element::ExampleBlock(inner) => inner,
+            Element::ExportBlock(inner) => inner,
+            Element::SrcBlock(inner) => inner,
+            Element::Clock(inner) => inner,
+            Element::DiarySexp(inner) => inner,
+            Element::Planning(inner) => inner,
+            Element::FixedWidthArea(inner) => inner,
+            Element::HorizontalRule(inner) => inner,
+            Element::Keyword(inner) => inner,
+            Element::BabelCall(inner) => inner,
+            Element::LatexEnvironment(inner) => inner,
+        }
+    }
+}
+
+impl<'s> GetElispFact<'s> for Object<'s> {
+    fn get_elisp_fact(&'s self) -> &'s dyn ElispFact<'s> {
+        match self {
+            Object::Bold(inner) => inner,
+            Object::Italic(inner) => inner,
+            Object::Underline(inner) => inner,
+            Object::StrikeThrough(inner) => inner,
+            Object::Code(inner) => inner,
+            Object::Verbatim(inner) => inner,
+            Object::PlainText(inner) => inner,
+            Object::RegularLink(inner) => inner,
+            Object::RadioLink(inner) => inner,
+            Object::RadioTarget(inner) => inner,
+            Object::PlainLink(inner) => inner,
+            Object::AngleLink(inner) => inner,
+            Object::OrgMacro(inner) => inner,
+            Object::Entity(inner) => inner,
+            Object::LatexFragment(inner) => inner,
+            Object::ExportSnippet(inner) => inner,
+            Object::FootnoteReference(inner) => inner,
+            Object::Citation(inner) => inner,
+            Object::CitationReference(inner) => inner,
+            Object::InlineBabelCall(inner) => inner,
+            Object::InlineSourceBlock(inner) => inner,
+            Object::LineBreak(inner) => inner,
+            Object::Target(inner) => inner,
+            Object::StatisticsCookie(inner) => inner,
+            Object::Subscript(inner) => inner,
+            Object::Superscript(inner) => inner,
+            Object::Timestamp(inner) => inner,
+        }
+    }
+}
+
+impl<'s> ElispFact<'s> for Document<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "org-data".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Section<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "section".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Heading<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "headline".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for PlainList<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "plain-list".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for PlainListItem<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "item".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for GreaterBlock<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        match self.name.to_lowercase().as_str() {
+            "center" => "center-block".into(),
+            "quote" => "quote-block".into(),
+            _ => "special-block".into(),
+        }
+    }
+}
+
+impl<'s> ElispFact<'s> for DynamicBlock<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "dynamic-block".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for FootnoteDefinition<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "footnote-definition".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Drawer<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "drawer".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for PropertyDrawer<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "property-drawer".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for NodeProperty<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "node-property".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Table<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "table".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for TableRow<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "table-row".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Paragraph<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "paragraph".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for TableCell<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "table-cell".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Comment<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "comment".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for VerseBlock<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "verse-block".into()
+    }
+}
+impl<'s> ElispFact<'s> for CommentBlock<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "comment-block".into()
+    }
+}
+impl<'s> ElispFact<'s> for ExampleBlock<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "example-block".into()
+    }
+}
+impl<'s> ElispFact<'s> for ExportBlock<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "export-block".into()
+    }
+}
+impl<'s> ElispFact<'s> for SrcBlock<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "src-block".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Clock<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "clock".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for DiarySexp<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "diary-sexp".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Planning<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "planning".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for FixedWidthArea<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "fixed-width".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for HorizontalRule<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "horizontal-rule".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Keyword<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "keyword".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for BabelCall<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "babel-call".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for LatexEnvironment<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "latex-environment".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Bold<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "bold".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Italic<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "italic".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Underline<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "underline".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for StrikeThrough<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "strike-through".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Code<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "code".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Verbatim<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "verbatim".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for RegularLink<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "link".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for RadioLink<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "link".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for RadioTarget<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "radio-target".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for PlainLink<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "link".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for AngleLink<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "link".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for OrgMacro<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "macro".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Entity<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "entity".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for LatexFragment<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "latex-fragment".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for ExportSnippet<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "export-snippet".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for FootnoteReference<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "footnote-reference".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Citation<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "citation".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for CitationReference<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "citation-reference".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for InlineBabelCall<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "inline-babel-call".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for InlineSourceBlock<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "inline-src-block".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for LineBreak<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "line-break".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Target<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "target".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for StatisticsCookie<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "statistics-cookie".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Subscript<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "subscript".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Superscript<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "superscript".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for Timestamp<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        "timestamp".into()
+    }
+}
+
+impl<'s> ElispFact<'s> for PlainText<'s> {
+    fn get_elisp_name(&'s self) -> Cow<'s, str> {
+        // plain text from upstream emacs does not actually have a name but this is included here to make rendering the status diff easier.
+        "plain-text".into()
+    }
+}

src/compare/mod.rs

@@ -1,5 +1,6 @@
 mod compare;
 mod diff;
+mod elisp_fact;
 mod parse;
 mod sexp;
 mod util;

src/compare/sexp.rs

@@ -106,8 +106,8 @@ fn is_slice_of(parent: &str, child: &str) -> bool {
 }
 
 /// Get a slice of the string that was consumed in a parser using the original input to the parser and the remaining input after the parser.
-pub fn get_consumed<'s>(input: &'s str, remaining: &'s str) -> &'s str {
-    assert!(is_slice_of(input, remaining));
+fn get_consumed<'s>(input: &'s str, remaining: &'s str) -> &'s str {
+    debug_assert!(is_slice_of(input, remaining));
     let source = {
         let offset = remaining.as_ptr() as usize - input.as_ptr() as usize;
         &input[..offset]

src/compare/util.rs

@@ -1,5 +1,7 @@
+use super::elisp_fact::GetElispFact;
 use super::sexp::Token;
-use crate::types::Source;
+use crate::types::GetStandardProperties;
+use crate::types::StandardProperties;
 
 /// Check if the child string slice is a slice of the parent string slice.
 fn is_slice_of(parent: &str, child: &str) -> bool {
@@ -10,21 +12,38 @@ fn is_slice_of(parent: &str, child: &str) -> bool {
     child_start >= parent_start && child_end <= parent_end
 }
 
-/// Get the offset into source that the rust object exists at.
+/// Get the byte offset into source that the rust object exists at.
 ///
 /// These offsets are zero-based unlike the elisp ones.
-fn get_offsets<'s, S: Source<'s>>(source: &'s str, rust_object: &'s S) -> (usize, usize) {
-    let rust_object_source = rust_object.get_source();
-    assert!(is_slice_of(source, rust_object_source));
-    let offset = rust_object_source.as_ptr() as usize - source.as_ptr() as usize;
+fn get_rust_byte_offsets<'s, S: StandardProperties<'s> + ?Sized>(
+    original_document: &'s str,
+    rust_ast_node: &'s S,
+) -> (usize, usize) {
+    let rust_object_source = rust_ast_node.get_source();
+    debug_assert!(is_slice_of(original_document, rust_object_source));
+    let offset = rust_object_source.as_ptr() as usize - original_document.as_ptr() as usize;
     let end = offset + rust_object_source.len();
     (offset, end)
 }
 
-pub(crate) fn assert_name<'s>(
+pub(crate) fn compare_standard_properties<
+    's,
+    S: GetStandardProperties<'s> + GetElispFact<'s> + ?Sized,
+>(
+    original_document: &'s str,
     emacs: &'s Token<'s>,
-    name: &str,
+    rust: &'s S,
 ) -> Result<(), Box<dyn std::error::Error>> {
+    assert_name(emacs, rust.get_elisp_fact().get_elisp_name())?;
+    assert_bounds(original_document, emacs, rust.get_standard_properties())?;
+    Ok(())
+}
+
+pub(crate) fn assert_name<'s, S: AsRef<str>>(
+    emacs: &'s Token<'s>,
+    name: S,
+) -> Result<(), Box<dyn std::error::Error>> {
+    let name = name.as_ref();
     let children = emacs.as_list()?;
     let first_child = children
         .first()
@@ -32,7 +51,7 @@ pub(crate) fn assert_name<'s>(
         .as_atom()?;
     if first_child != name {
         Err(format!(
-            "Expected a {expected} cell, but found a {found} cell.",
+            "AST node name mismatch. Expected a (rust) {expected} cell, but found a (emacs) {found} cell.",
             expected = name,
             found = first_child
         ))?;
@@ -40,30 +59,33 @@ pub(crate) fn assert_name<'s>(
     Ok(())
 }
 
-pub(crate) fn assert_bounds<'s, S: Source<'s>>(
-    source: &'s str,
+/// Assert that the character ranges defined by upstream org-mode's :standard-properties match the slices in Organic's StandardProperties.
+///
+/// This does **not** handle plain text because plain text is a special case.
+pub(crate) fn assert_bounds<'s, S: StandardProperties<'s> + ?Sized>(
+    original_document: &'s str,
     emacs: &'s Token<'s>,
     rust: &'s S,
 ) -> Result<(), Box<dyn std::error::Error>> {
-    let standard_properties = get_standard_properties(emacs)?;
+    let standard_properties = get_emacs_standard_properties(emacs)?; // 1-based
     let (begin, end) = (
         standard_properties
             .begin
             .ok_or("Token should have a begin.")?,
         standard_properties.end.ok_or("Token should have an end.")?,
     );
-    let (rust_begin, rust_end) = get_offsets(source, rust);
-    let rust_begin_char_offset = (&source[..rust_begin]).chars().count();
+    let (rust_begin, rust_end) = get_rust_byte_offsets(original_document, rust); // 0-based
+    let rust_begin_char_offset = (&original_document[..rust_begin]).chars().count() + 1; // 1-based
     let rust_end_char_offset =
-        rust_begin_char_offset + (&source[rust_begin..rust_end]).chars().count();
-    if (rust_begin_char_offset + 1) != begin || (rust_end_char_offset + 1) != end {
-        Err(format!("Rust bounds (in chars) ({rust_begin}, {rust_end}) do not match emacs bounds ({emacs_begin}, {emacs_end})", rust_begin = rust_begin_char_offset + 1, rust_end = rust_end_char_offset + 1, emacs_begin=begin, emacs_end=end))?;
+        rust_begin_char_offset + (&original_document[rust_begin..rust_end]).chars().count(); // 1-based
+    if rust_begin_char_offset != begin || rust_end_char_offset != end {
+        Err(format!("Rust bounds (in chars) ({rust_begin}, {rust_end}) do not match emacs bounds ({emacs_begin}, {emacs_end})", rust_begin = rust_begin_char_offset, rust_end = rust_end_char_offset, emacs_begin=begin, emacs_end=end))?;
     }
 
     Ok(())
 }
 
-struct StandardProperties {
+struct EmacsStandardProperties {
     begin: Option<usize>,
     #[allow(dead_code)]
     post_affiliated: Option<usize>,
@@ -76,9 +98,9 @@ struct StandardProperties {
     post_blank: Option<usize>,
 }
 
-fn get_standard_properties<'s>(
+fn get_emacs_standard_properties<'s>(
     emacs: &'s Token<'s>,
-) -> Result<StandardProperties, Box<dyn std::error::Error>> {
+) -> Result<EmacsStandardProperties, Box<dyn std::error::Error>> {
     let children = emacs.as_list()?;
     let attributes_child = children
         .iter()
@@ -97,7 +119,7 @@ fn get_standard_properties<'s>(
         let contents_end = maybe_token_to_usize(std_props.next())?;
         let end = maybe_token_to_usize(std_props.next())?;
         let post_blank = maybe_token_to_usize(std_props.next())?;
-        StandardProperties {
+        EmacsStandardProperties {
             begin,
             post_affiliated,
             contents_begin,
@@ -116,7 +138,7 @@ fn get_standard_properties<'s>(
             maybe_token_to_usize(attributes_map.get(":post-blank").map(|token| *token))?;
         let post_affiliated =
             maybe_token_to_usize(attributes_map.get(":post-affiliated").map(|token| *token))?;
-        StandardProperties {
+        EmacsStandardProperties {
             begin,
             post_affiliated,
             contents_begin,

src/context/list.rs

@@ -6,6 +6,7 @@ pub(crate) struct List<'parent, T> {
     parent: Link<'parent, T>,
 }
 
+// TODO: Should I be defining a lifetime for T in the generics here? Ref: https://quinedot.github.io/rust-learning/dyn-elision-advanced.html#iteraction-with-type-aliases
 type Link<'parent, T> = Option<&'parent List<'parent, T>>;
 
 impl<'parent, T> List<'parent, T> {

src/error/error.rs

@@ -4,10 +4,9 @@ use nom::IResult;
 
 pub(crate) type Res<T, U> = IResult<T, U, CustomError<T>>;
 
-// TODO: MyError probably shouldn't be based on the same type as the input type since it's used exclusively with static strings right now.
 #[derive(Debug)]
 pub enum CustomError<I> {
-    MyError(MyError<I>),
+    MyError(MyError<&'static str>),
     Nom(I, ErrorKind),
     IO(std::io::Error),
 }
@@ -31,3 +30,9 @@ impl<I> From<std::io::Error> for CustomError<I> {
         CustomError::IO(value)
     }
 }
+
+impl<I> From<&'static str> for CustomError<I> {
+    fn from(value: &'static str) -> Self {
+        CustomError::MyError(MyError(value))
+    }
+}

src/iter/all_ast_node_iter.rs

@@ -0,0 +1,102 @@
+use std::collections::VecDeque;
+
+use super::ast_node::AstNode;
+use super::ast_node_iter::AstNodeIter;
+
+pub struct AllAstNodeIter<'r, 's> {
+    root: Option<AstNode<'r, 's>>,
+    queue: VecDeque<AstNodeIter<'r, 's>>,
+}
+
+impl<'r, 's> Iterator for AllAstNodeIter<'r, 's> {
+    type Item = AstNode<'r, 's>;
+
+    fn next(&mut self) -> Option<Self::Item> {
+        if let Some(root) = self.root.take() {
+            self.queue.push_back(AstNodeIter::from_ast_node(&root));
+            return Some(root);
+        }
+        while let Some(child) = self.queue.front_mut() {
+            let next_elem_this_iter = match child {
+                AstNodeIter::Document(ref mut i) => i.next(),
+                AstNodeIter::Heading(ref mut i) => i.next(),
+                AstNodeIter::Section(ref mut i) => i.next(),
+                AstNodeIter::Paragraph(ref mut i) => i.next(),
+                AstNodeIter::PlainList(ref mut i) => i.next(),
+                AstNodeIter::PlainListItem(ref mut i) => i.next(),
+                AstNodeIter::GreaterBlock(ref mut i) => i.next(),
+                AstNodeIter::DynamicBlock(ref mut i) => i.next(),
+                AstNodeIter::FootnoteDefinition(ref mut i) => i.next(),
+                AstNodeIter::Comment(ref mut i) => i.next(),
+                AstNodeIter::Drawer(ref mut i) => i.next(),
+                AstNodeIter::PropertyDrawer(ref mut i) => i.next(),
+                AstNodeIter::NodeProperty(ref mut i) => i.next(),
+                AstNodeIter::Table(ref mut i) => i.next(),
+                AstNodeIter::TableRow(ref mut i) => i.next(),
+                AstNodeIter::VerseBlock(ref mut i) => i.next(),
+                AstNodeIter::CommentBlock(ref mut i) => i.next(),
+                AstNodeIter::ExampleBlock(ref mut i) => i.next(),
+                AstNodeIter::ExportBlock(ref mut i) => i.next(),
+                AstNodeIter::SrcBlock(ref mut i) => i.next(),
+                AstNodeIter::Clock(ref mut i) => i.next(),
+                AstNodeIter::DiarySexp(ref mut i) => i.next(),
+                AstNodeIter::Planning(ref mut i) => i.next(),
+                AstNodeIter::FixedWidthArea(ref mut i) => i.next(),
+                AstNodeIter::HorizontalRule(ref mut i) => i.next(),
+                AstNodeIter::Keyword(ref mut i) => i.next(),
+                AstNodeIter::BabelCall(ref mut i) => i.next(),
+                AstNodeIter::LatexEnvironment(ref mut i) => i.next(),
+                AstNodeIter::Bold(ref mut i) => i.next(),
+                AstNodeIter::Italic(ref mut i) => i.next(),
+                AstNodeIter::Underline(ref mut i) => i.next(),
+                AstNodeIter::StrikeThrough(ref mut i) => i.next(),
+                AstNodeIter::Code(ref mut i) => i.next(),
+                AstNodeIter::Verbatim(ref mut i) => i.next(),
+                AstNodeIter::PlainText(ref mut i) => i.next(),
+                AstNodeIter::RegularLink(ref mut i) => i.next(),
+                AstNodeIter::RadioLink(ref mut i) => i.next(),
+                AstNodeIter::RadioTarget(ref mut i) => i.next(),
+                AstNodeIter::PlainLink(ref mut i) => i.next(),
+                AstNodeIter::AngleLink(ref mut i) => i.next(),
+                AstNodeIter::OrgMacro(ref mut i) => i.next(),
+                AstNodeIter::Entity(ref mut i) => i.next(),
+                AstNodeIter::LatexFragment(ref mut i) => i.next(),
+                AstNodeIter::ExportSnippet(ref mut i) => i.next(),
+                AstNodeIter::FootnoteReference(ref mut i) => i.next(),
+                AstNodeIter::Citation(ref mut i) => i.next(),
+                AstNodeIter::CitationReference(ref mut i) => i.next(),
+                AstNodeIter::InlineBabelCall(ref mut i) => i.next(),
+                AstNodeIter::InlineSourceBlock(ref mut i) => i.next(),
+                AstNodeIter::LineBreak(ref mut i) => i.next(),
+                AstNodeIter::Target(ref mut i) => i.next(),
+                AstNodeIter::StatisticsCookie(ref mut i) => i.next(),
+                AstNodeIter::Subscript(ref mut i) => i.next(),
+                AstNodeIter::Superscript(ref mut i) => i.next(),
+                AstNodeIter::TableCell(ref mut i) => i.next(),
+                AstNodeIter::Timestamp(ref mut i) => i.next(),
+            };
+            if let Some(next_elem_this_iter) = next_elem_this_iter {
+                self.queue
+                    .push_back(AstNodeIter::from_ast_node(&next_elem_this_iter));
+                return Some(next_elem_this_iter);
+            } else {
+                self.queue.pop_front();
+            }
+        }
+
+        None
+    }
+}
+
+impl<'r, 's> IntoIterator for AstNode<'r, 's> {
+    type Item = AstNode<'r, 's>;
+
+    type IntoIter = AllAstNodeIter<'r, 's>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        AllAstNodeIter {
+            root: Some(self),
+            queue: VecDeque::new(),
+        }
+    }
+}

src/iter/ast_node.rs

@@ -0,0 +1,251 @@
+use super::macros::to_ast_node;
+use crate::types::AngleLink;
+use crate::types::BabelCall;
+use crate::types::Bold;
+use crate::types::Citation;
+use crate::types::CitationReference;
+use crate::types::Clock;
+use crate::types::Code;
+use crate::types::Comment;
+use crate::types::CommentBlock;
+use crate::types::DiarySexp;
+use crate::types::Document;
+use crate::types::DocumentElement;
+use crate::types::Drawer;
+use crate::types::DynamicBlock;
+use crate::types::Element;
+use crate::types::Entity;
+use crate::types::ExampleBlock;
+use crate::types::ExportBlock;
+use crate::types::ExportSnippet;
+use crate::types::FixedWidthArea;
+use crate::types::FootnoteDefinition;
+use crate::types::FootnoteReference;
+use crate::types::GreaterBlock;
+use crate::types::Heading;
+use crate::types::HorizontalRule;
+use crate::types::InlineBabelCall;
+use crate::types::InlineSourceBlock;
+use crate::types::Italic;
+use crate::types::Keyword;
+use crate::types::LatexEnvironment;
+use crate::types::LatexFragment;
+use crate::types::LineBreak;
+use crate::types::NodeProperty;
+use crate::types::Object;
+use crate::types::OrgMacro;
+use crate::types::Paragraph;
+use crate::types::PlainLink;
+use crate::types::PlainList;
+use crate::types::PlainListItem;
+use crate::types::PlainText;
+use crate::types::Planning;
+use crate::types::PropertyDrawer;
+use crate::types::RadioLink;
+use crate::types::RadioTarget;
+use crate::types::RegularLink;
+use crate::types::Section;
+use crate::types::SrcBlock;
+use crate::types::StatisticsCookie;
+use crate::types::StrikeThrough;
+use crate::types::Subscript;
+use crate::types::Superscript;
+use crate::types::Table;
+use crate::types::TableCell;
+use crate::types::TableRow;
+use crate::types::Target;
+use crate::types::Timestamp;
+use crate::types::Underline;
+use crate::types::Verbatim;
+use crate::types::VerseBlock;
+
+pub enum AstNode<'r, 's> {
+    // Document Nodes
+    Document(&'r Document<'s>),
+    Heading(&'r Heading<'s>),
+    Section(&'r Section<'s>),
+    // Elements
+    Paragraph(&'r Paragraph<'s>),
+    PlainList(&'r PlainList<'s>),
+    PlainListItem(&'r PlainListItem<'s>),
+    GreaterBlock(&'r GreaterBlock<'s>),
+    DynamicBlock(&'r DynamicBlock<'s>),
+    FootnoteDefinition(&'r FootnoteDefinition<'s>),
+    Comment(&'r Comment<'s>),
+    Drawer(&'r Drawer<'s>),
+    PropertyDrawer(&'r PropertyDrawer<'s>),
+    NodeProperty(&'r NodeProperty<'s>),
+    Table(&'r Table<'s>),
+    TableRow(&'r TableRow<'s>),
+    VerseBlock(&'r VerseBlock<'s>),
+    CommentBlock(&'r CommentBlock<'s>),
+    ExampleBlock(&'r ExampleBlock<'s>),
+    ExportBlock(&'r ExportBlock<'s>),
+    SrcBlock(&'r SrcBlock<'s>),
+    Clock(&'r Clock<'s>),
+    DiarySexp(&'r DiarySexp<'s>),
+    Planning(&'r Planning<'s>),
+    FixedWidthArea(&'r FixedWidthArea<'s>),
+    HorizontalRule(&'r HorizontalRule<'s>),
+    Keyword(&'r Keyword<'s>),
+    BabelCall(&'r BabelCall<'s>),
+    LatexEnvironment(&'r LatexEnvironment<'s>),
+    // Objects
+    Bold(&'r Bold<'s>),
+    Italic(&'r Italic<'s>),
+    Underline(&'r Underline<'s>),
+    StrikeThrough(&'r StrikeThrough<'s>),
+    Code(&'r Code<'s>),
+    Verbatim(&'r Verbatim<'s>),
+    PlainText(&'r PlainText<'s>),
+    RegularLink(&'r RegularLink<'s>),
+    RadioLink(&'r RadioLink<'s>),
+    RadioTarget(&'r RadioTarget<'s>),
+    PlainLink(&'r PlainLink<'s>),
+    AngleLink(&'r AngleLink<'s>),
+    OrgMacro(&'r OrgMacro<'s>),
+    Entity(&'r Entity<'s>),
+    LatexFragment(&'r LatexFragment<'s>),
+    ExportSnippet(&'r ExportSnippet<'s>),
+    FootnoteReference(&'r FootnoteReference<'s>),
+    Citation(&'r Citation<'s>),
+    CitationReference(&'r CitationReference<'s>),
+    InlineBabelCall(&'r InlineBabelCall<'s>),
+    InlineSourceBlock(&'r InlineSourceBlock<'s>),
+    LineBreak(&'r LineBreak<'s>),
+    Target(&'r Target<'s>),
+    StatisticsCookie(&'r StatisticsCookie<'s>),
+    Subscript(&'r Subscript<'s>),
+    Superscript(&'r Superscript<'s>),
+    TableCell(&'r TableCell<'s>),
+    Timestamp(&'r Timestamp<'s>),
+}
+
+impl<'r, 's> From<&'r DocumentElement<'s>> for AstNode<'r, 's> {
+    fn from(value: &'r DocumentElement<'s>) -> Self {
+        match value {
+            DocumentElement::Heading(inner) => inner.into(),
+            DocumentElement::Section(inner) => inner.into(),
+        }
+    }
+}
+
+impl<'r, 's> From<&'r Element<'s>> for AstNode<'r, 's> {
+    fn from(value: &'r Element<'s>) -> Self {
+        match value {
+            Element::Paragraph(inner) => inner.into(),
+            Element::PlainList(inner) => inner.into(),
+            Element::GreaterBlock(inner) => inner.into(),
+            Element::DynamicBlock(inner) => inner.into(),
+            Element::FootnoteDefinition(inner) => inner.into(),
+            Element::Comment(inner) => inner.into(),
+            Element::Drawer(inner) => inner.into(),
+            Element::PropertyDrawer(inner) => inner.into(),
+            Element::Table(inner) => inner.into(),
+            Element::VerseBlock(inner) => inner.into(),
+            Element::CommentBlock(inner) => inner.into(),
+            Element::ExampleBlock(inner) => inner.into(),
+            Element::ExportBlock(inner) => inner.into(),
+            Element::SrcBlock(inner) => inner.into(),
+            Element::Clock(inner) => inner.into(),
+            Element::DiarySexp(inner) => inner.into(),
+            Element::Planning(inner) => inner.into(),
+            Element::FixedWidthArea(inner) => inner.into(),
+            Element::HorizontalRule(inner) => inner.into(),
+            Element::Keyword(inner) => inner.into(),
+            Element::BabelCall(inner) => inner.into(),
+            Element::LatexEnvironment(inner) => inner.into(),
+        }
+    }
+}
+
+impl<'r, 's> From<&'r Object<'s>> for AstNode<'r, 's> {
+    fn from(value: &'r Object<'s>) -> Self {
+        match value {
+            Object::Bold(inner) => inner.into(),
+            Object::Italic(inner) => inner.into(),
+            Object::Underline(inner) => inner.into(),
+            Object::StrikeThrough(inner) => inner.into(),
+            Object::Code(inner) => inner.into(),
+            Object::Verbatim(inner) => inner.into(),
+            Object::PlainText(inner) => inner.into(),
+            Object::RegularLink(inner) => inner.into(),
+            Object::RadioLink(inner) => inner.into(),
+            Object::RadioTarget(inner) => inner.into(),
+            Object::PlainLink(inner) => inner.into(),
+            Object::AngleLink(inner) => inner.into(),
+            Object::OrgMacro(inner) => inner.into(),
+            Object::Entity(inner) => inner.into(),
+            Object::LatexFragment(inner) => inner.into(),
+            Object::ExportSnippet(inner) => inner.into(),
+            Object::FootnoteReference(inner) => inner.into(),
+            Object::Citation(inner) => inner.into(),
+            Object::CitationReference(inner) => inner.into(),
+            Object::InlineBabelCall(inner) => inner.into(),
+            Object::InlineSourceBlock(inner) => inner.into(),
+            Object::LineBreak(inner) => inner.into(),
+            Object::Target(inner) => inner.into(),
+            Object::StatisticsCookie(inner) => inner.into(),
+            Object::Subscript(inner) => inner.into(),
+            Object::Superscript(inner) => inner.into(),
+            Object::Timestamp(inner) => inner.into(),
+        }
+    }
+}
+
+to_ast_node!(&'r Document<'s>, AstNode::Document);
+to_ast_node!(&'r Heading<'s>, AstNode::Heading);
+to_ast_node!(&'r Section<'s>, AstNode::Section);
+to_ast_node!(&'r Paragraph<'s>, AstNode::Paragraph);
+to_ast_node!(&'r PlainList<'s>, AstNode::PlainList);
+to_ast_node!(&'r PlainListItem<'s>, AstNode::PlainListItem);
+to_ast_node!(&'r GreaterBlock<'s>, AstNode::GreaterBlock);
+to_ast_node!(&'r DynamicBlock<'s>, AstNode::DynamicBlock);
+to_ast_node!(&'r FootnoteDefinition<'s>, AstNode::FootnoteDefinition);
+to_ast_node!(&'r Comment<'s>, AstNode::Comment);
+to_ast_node!(&'r Drawer<'s>, AstNode::Drawer);
+to_ast_node!(&'r PropertyDrawer<'s>, AstNode::PropertyDrawer);
+to_ast_node!(&'r NodeProperty<'s>, AstNode::NodeProperty);
+to_ast_node!(&'r Table<'s>, AstNode::Table);
+to_ast_node!(&'r TableRow<'s>, AstNode::TableRow);
+to_ast_node!(&'r VerseBlock<'s>, AstNode::VerseBlock);
+to_ast_node!(&'r CommentBlock<'s>, AstNode::CommentBlock);
+to_ast_node!(&'r ExampleBlock<'s>, AstNode::ExampleBlock);
+to_ast_node!(&'r ExportBlock<'s>, AstNode::ExportBlock);
+to_ast_node!(&'r SrcBlock<'s>, AstNode::SrcBlock);
+to_ast_node!(&'r Clock<'s>, AstNode::Clock);
+to_ast_node!(&'r DiarySexp<'s>, AstNode::DiarySexp);
+to_ast_node!(&'r Planning<'s>, AstNode::Planning);
+to_ast_node!(&'r FixedWidthArea<'s>, AstNode::FixedWidthArea);
+to_ast_node!(&'r HorizontalRule<'s>, AstNode::HorizontalRule);
+to_ast_node!(&'r Keyword<'s>, AstNode::Keyword);
+to_ast_node!(&'r BabelCall<'s>, AstNode::BabelCall);
+to_ast_node!(&'r LatexEnvironment<'s>, AstNode::LatexEnvironment);
+to_ast_node!(&'r Bold<'s>, AstNode::Bold);
+to_ast_node!(&'r Italic<'s>, AstNode::Italic);
+to_ast_node!(&'r Underline<'s>, AstNode::Underline);
+to_ast_node!(&'r StrikeThrough<'s>, AstNode::StrikeThrough);
+to_ast_node!(&'r Code<'s>, AstNode::Code);
+to_ast_node!(&'r Verbatim<'s>, AstNode::Verbatim);
+to_ast_node!(&'r PlainText<'s>, AstNode::PlainText);
+to_ast_node!(&'r RegularLink<'s>, AstNode::RegularLink);
+to_ast_node!(&'r RadioLink<'s>, AstNode::RadioLink);
+to_ast_node!(&'r RadioTarget<'s>, AstNode::RadioTarget);
+to_ast_node!(&'r PlainLink<'s>, AstNode::PlainLink);
+to_ast_node!(&'r AngleLink<'s>, AstNode::AngleLink);
+to_ast_node!(&'r OrgMacro<'s>, AstNode::OrgMacro);
+to_ast_node!(&'r Entity<'s>, AstNode::Entity);
+to_ast_node!(&'r LatexFragment<'s>, AstNode::LatexFragment);
+to_ast_node!(&'r ExportSnippet<'s>, AstNode::ExportSnippet);
+to_ast_node!(&'r FootnoteReference<'s>, AstNode::FootnoteReference);
+to_ast_node!(&'r Citation<'s>, AstNode::Citation);
+to_ast_node!(&'r CitationReference<'s>, AstNode::CitationReference);
+to_ast_node!(&'r InlineBabelCall<'s>, AstNode::InlineBabelCall);
+to_ast_node!(&'r InlineSourceBlock<'s>, AstNode::InlineSourceBlock);
+to_ast_node!(&'r LineBreak<'s>, AstNode::LineBreak);
+to_ast_node!(&'r Target<'s>, AstNode::Target);
+to_ast_node!(&'r StatisticsCookie<'s>, AstNode::StatisticsCookie);
+to_ast_node!(&'r Subscript<'s>, AstNode::Subscript);
+to_ast_node!(&'r Superscript<'s>, AstNode::Superscript);
+to_ast_node!(&'r TableCell<'s>, AstNode::TableCell);
+to_ast_node!(&'r Timestamp<'s>, AstNode::Timestamp);

src/iter/ast_node_iter.rs

@@ -0,0 +1,332 @@
+use std::marker::PhantomData;
+
+use super::ast_node::AstNode;
+use super::macros::children_iter;
+use super::macros::empty_iter;
+use super::macros::multi_field_iter;
+use crate::types::AngleLink;
+use crate::types::BabelCall;
+use crate::types::Bold;
+use crate::types::Citation;
+use crate::types::CitationReference;
+use crate::types::Clock;
+use crate::types::Code;
+use crate::types::Comment;
+use crate::types::CommentBlock;
+use crate::types::DiarySexp;
+use crate::types::Document;
+use crate::types::DocumentElement;
+use crate::types::Drawer;
+use crate::types::DynamicBlock;
+use crate::types::Element;
+use crate::types::Entity;
+use crate::types::ExampleBlock;
+use crate::types::ExportBlock;
+use crate::types::ExportSnippet;
+use crate::types::FixedWidthArea;
+use crate::types::FootnoteDefinition;
+use crate::types::FootnoteReference;
+use crate::types::GreaterBlock;
+use crate::types::Heading;
+use crate::types::HorizontalRule;
+use crate::types::InlineBabelCall;
+use crate::types::InlineSourceBlock;
+use crate::types::Italic;
+use crate::types::Keyword;
+use crate::types::LatexEnvironment;
+use crate::types::LatexFragment;
+use crate::types::LineBreak;
+use crate::types::NodeProperty;
+use crate::types::Object;
+use crate::types::OrgMacro;
+use crate::types::Paragraph;
+use crate::types::PlainLink;
+use crate::types::PlainList;
+use crate::types::PlainListItem;
+use crate::types::PlainText;
+use crate::types::Planning;
+use crate::types::PropertyDrawer;
+use crate::types::RadioLink;
+use crate::types::RadioTarget;
+use crate::types::RegularLink;
+use crate::types::Section;
+use crate::types::SrcBlock;
+use crate::types::StatisticsCookie;
+use crate::types::StrikeThrough;
+use crate::types::Subscript;
+use crate::types::Superscript;
+use crate::types::Table;
+use crate::types::TableCell;
+use crate::types::TableRow;
+use crate::types::Target;
+use crate::types::Timestamp;
+use crate::types::Underline;
+use crate::types::Verbatim;
+use crate::types::VerseBlock;
+
+/// Iterator over the AST nodes contained within the starting node.
+///
+/// This only iterates over the children, not the starting node itself. So an AstNodeIter::PlainList would only return PlainListItems, not the PlainList.
+///
+/// This only iterates over AST nodes, so an AstNodeIter::Heading would iterate over both the title and section contents, but it would not iterate over simple strings like the TODO keyword or priority.
+pub(crate) enum AstNodeIter<'r, 's> {
+    // Document Nodes
+    Document(DocumentIter<'r, 's>),
+    Heading(HeadingIter<'r, 's>),
+    Section(SectionIter<'r, 's>),
+    // Elements
+    Paragraph(ParagraphIter<'r, 's>),
+    PlainList(PlainListIter<'r, 's>),
+    PlainListItem(PlainListItemIter<'r, 's>),
+    GreaterBlock(GreaterBlockIter<'r, 's>),
+    DynamicBlock(DynamicBlockIter<'r, 's>),
+    FootnoteDefinition(FootnoteDefinitionIter<'r, 's>),
+    Comment(CommentIter<'r, 's>),
+    Drawer(DrawerIter<'r, 's>),
+    PropertyDrawer(PropertyDrawerIter<'r, 's>),
+    NodeProperty(NodePropertyIter<'r, 's>),
+    Table(TableIter<'r, 's>),
+    TableRow(TableRowIter<'r, 's>),
+    VerseBlock(VerseBlockIter<'r, 's>),
+    CommentBlock(CommentBlockIter<'r, 's>),
+    ExampleBlock(ExampleBlockIter<'r, 's>),
+    ExportBlock(ExportBlockIter<'r, 's>),
+    SrcBlock(SrcBlockIter<'r, 's>),
+    Clock(ClockIter<'r, 's>),
+    DiarySexp(DiarySexpIter<'r, 's>),
+    Planning(PlanningIter<'r, 's>),
+    FixedWidthArea(FixedWidthAreaIter<'r, 's>),
+    HorizontalRule(HorizontalRuleIter<'r, 's>),
+    Keyword(KeywordIter<'r, 's>),
+    BabelCall(BabelCallIter<'r, 's>),
+    LatexEnvironment(LatexEnvironmentIter<'r, 's>),
+    // Objects
+    Bold(BoldIter<'r, 's>),
+    Italic(ItalicIter<'r, 's>),
+    Underline(UnderlineIter<'r, 's>),
+    StrikeThrough(StrikeThroughIter<'r, 's>),
+    Code(CodeIter<'r, 's>),
+    Verbatim(VerbatimIter<'r, 's>),
+    PlainText(PlainTextIter<'r, 's>),
+    RegularLink(RegularLinkIter<'r, 's>),
+    RadioLink(RadioLinkIter<'r, 's>),
+    RadioTarget(RadioTargetIter<'r, 's>),
+    PlainLink(PlainLinkIter<'r, 's>),
+    AngleLink(AngleLinkIter<'r, 's>),
+    OrgMacro(OrgMacroIter<'r, 's>),
+    Entity(EntityIter<'r, 's>),
+    LatexFragment(LatexFragmentIter<'r, 's>),
+    ExportSnippet(ExportSnippetIter<'r, 's>),
+    FootnoteReference(FootnoteReferenceIter<'r, 's>),
+    Citation(CitationIter<'r, 's>),
+    CitationReference(CitationReferenceIter<'r, 's>),
+    InlineBabelCall(InlineBabelCallIter<'r, 's>),
+    InlineSourceBlock(InlineSourceBlockIter<'r, 's>),
+    LineBreak(LineBreakIter<'r, 's>),
+    Target(TargetIter<'r, 's>),
+    StatisticsCookie(StatisticsCookieIter<'r, 's>),
+    Subscript(SubscriptIter<'r, 's>),
+    Superscript(SuperscriptIter<'r, 's>),
+    TableCell(TableCellIter<'r, 's>),
+    Timestamp(TimestampIter<'r, 's>),
+}
+
+impl<'r, 's> AstNodeIter<'r, 's> {
+    pub(crate) fn from_ast_node(node: &AstNode<'r, 's>) -> AstNodeIter<'r, 's> {
+        match node {
+            AstNode::Document(inner) => AstNodeIter::Document(inner.into_iter()),
+            AstNode::Heading(inner) => AstNodeIter::Heading(inner.into_iter()),
+            AstNode::Section(inner) => AstNodeIter::Section(inner.into_iter()),
+            AstNode::Paragraph(inner) => AstNodeIter::Paragraph(inner.into_iter()),
+            AstNode::PlainList(inner) => AstNodeIter::PlainList(inner.into_iter()),
+            AstNode::PlainListItem(inner) => AstNodeIter::PlainListItem(inner.into_iter()),
+            AstNode::GreaterBlock(inner) => AstNodeIter::GreaterBlock(inner.into_iter()),
+            AstNode::DynamicBlock(inner) => AstNodeIter::DynamicBlock(inner.into_iter()),
+            AstNode::FootnoteDefinition(inner) => {
+                AstNodeIter::FootnoteDefinition(inner.into_iter())
+            }
+            AstNode::Comment(inner) => AstNodeIter::Comment(inner.into_iter()),
+            AstNode::Drawer(inner) => AstNodeIter::Drawer(inner.into_iter()),
+            AstNode::PropertyDrawer(inner) => AstNodeIter::PropertyDrawer(inner.into_iter()),
+            AstNode::NodeProperty(inner) => AstNodeIter::NodeProperty(inner.into_iter()),
+            AstNode::Table(inner) => AstNodeIter::Table(inner.into_iter()),
+            AstNode::TableRow(inner) => AstNodeIter::TableRow(inner.into_iter()),
+            AstNode::VerseBlock(inner) => AstNodeIter::VerseBlock(inner.into_iter()),
+            AstNode::CommentBlock(inner) => AstNodeIter::CommentBlock(inner.into_iter()),
+            AstNode::ExampleBlock(inner) => AstNodeIter::ExampleBlock(inner.into_iter()),
+            AstNode::ExportBlock(inner) => AstNodeIter::ExportBlock(inner.into_iter()),
+            AstNode::SrcBlock(inner) => AstNodeIter::SrcBlock(inner.into_iter()),
+            AstNode::Clock(inner) => AstNodeIter::Clock(inner.into_iter()),
+            AstNode::DiarySexp(inner) => AstNodeIter::DiarySexp(inner.into_iter()),
+            AstNode::Planning(inner) => AstNodeIter::Planning(inner.into_iter()),
+            AstNode::FixedWidthArea(inner) => AstNodeIter::FixedWidthArea(inner.into_iter()),
+            AstNode::HorizontalRule(inner) => AstNodeIter::HorizontalRule(inner.into_iter()),
+            AstNode::Keyword(inner) => AstNodeIter::Keyword(inner.into_iter()),
+            AstNode::BabelCall(inner) => AstNodeIter::BabelCall(inner.into_iter()),
+            AstNode::LatexEnvironment(inner) => AstNodeIter::LatexEnvironment(inner.into_iter()),
+            AstNode::Bold(inner) => AstNodeIter::Bold(inner.into_iter()),
+            AstNode::Italic(inner) => AstNodeIter::Italic(inner.into_iter()),
+            AstNode::Underline(inner) => AstNodeIter::Underline(inner.into_iter()),
+            AstNode::StrikeThrough(inner) => AstNodeIter::StrikeThrough(inner.into_iter()),
+            AstNode::Code(inner) => AstNodeIter::Code(inner.into_iter()),
+            AstNode::Verbatim(inner) => AstNodeIter::Verbatim(inner.into_iter()),
+            AstNode::PlainText(inner) => AstNodeIter::PlainText(inner.into_iter()),
+            AstNode::RegularLink(inner) => AstNodeIter::RegularLink(inner.into_iter()),
+            AstNode::RadioLink(inner) => AstNodeIter::RadioLink(inner.into_iter()),
+            AstNode::RadioTarget(inner) => AstNodeIter::RadioTarget(inner.into_iter()),
+            AstNode::PlainLink(inner) => AstNodeIter::PlainLink(inner.into_iter()),
+            AstNode::AngleLink(inner) => AstNodeIter::AngleLink(inner.into_iter()),
+            AstNode::OrgMacro(inner) => AstNodeIter::OrgMacro(inner.into_iter()),
+            AstNode::Entity(inner) => AstNodeIter::Entity(inner.into_iter()),
+            AstNode::LatexFragment(inner) => AstNodeIter::LatexFragment(inner.into_iter()),
+            AstNode::ExportSnippet(inner) => AstNodeIter::ExportSnippet(inner.into_iter()),
+            AstNode::FootnoteReference(inner) => AstNodeIter::FootnoteReference(inner.into_iter()),
+            AstNode::Citation(inner) => AstNodeIter::Citation(inner.into_iter()),
+            AstNode::CitationReference(inner) => AstNodeIter::CitationReference(inner.into_iter()),
+            AstNode::InlineBabelCall(inner) => AstNodeIter::InlineBabelCall(inner.into_iter()),
+            AstNode::InlineSourceBlock(inner) => AstNodeIter::InlineSourceBlock(inner.into_iter()),
+            AstNode::LineBreak(inner) => AstNodeIter::LineBreak(inner.into_iter()),
+            AstNode::Target(inner) => AstNodeIter::Target(inner.into_iter()),
+            AstNode::StatisticsCookie(inner) => AstNodeIter::StatisticsCookie(inner.into_iter()),
+            AstNode::Subscript(inner) => AstNodeIter::Subscript(inner.into_iter()),
+            AstNode::Superscript(inner) => AstNodeIter::Superscript(inner.into_iter()),
+            AstNode::TableCell(inner) => AstNodeIter::TableCell(inner.into_iter()),
+            AstNode::Timestamp(inner) => AstNodeIter::Timestamp(inner.into_iter()),
+        }
+    }
+}
+
+multi_field_iter!(
+    Document<'s>,
+    DocumentIter,
+    zeroth_section,
+    std::option::Iter<'r, Section<'s>>,
+    children,
+    std::slice::Iter<'r, Heading<'s>>
+);
+multi_field_iter!(
+    Heading<'s>,
+    HeadingIter,
+    title,
+    std::slice::Iter<'r, Object<'s>>,
+    children,
+    std::slice::Iter<'r, DocumentElement<'s>>
+);
+children_iter!(Section<'s>, SectionIter, std::slice::Iter<'r, Element<'s>>);
+children_iter!(
+    Paragraph<'s>,
+    ParagraphIter,
+    std::slice::Iter<'r, Object<'s>>
+);
+children_iter!(
+    PlainList<'s>,
+    PlainListIter,
+    std::slice::Iter<'r, PlainListItem<'s>>
+);
+multi_field_iter!(
+    PlainListItem<'s>,
+    PlainListItemIter,
+    tag,
+    std::slice::Iter<'r, Object<'s>>,
+    children,
+    std::slice::Iter<'r, Element<'s>>
+);
+children_iter!(
+    GreaterBlock<'s>,
+    GreaterBlockIter,
+    std::slice::Iter<'r, Element<'s>>
+);
+children_iter!(
+    DynamicBlock<'s>,
+    DynamicBlockIter,
+    std::slice::Iter<'r, Element<'s>>
+);
+children_iter!(
+    FootnoteDefinition<'s>,
+    FootnoteDefinitionIter,
+    std::slice::Iter<'r, Element<'s>>
+);
+empty_iter!(Comment<'s>, CommentIter);
+children_iter!(Drawer<'s>, DrawerIter, std::slice::Iter<'r, Element<'s>>);
+children_iter!(
+    PropertyDrawer<'s>,
+    PropertyDrawerIter,
+    std::slice::Iter<'r, NodeProperty<'s>>
+);
+empty_iter!(NodeProperty<'s>, NodePropertyIter);
+children_iter!(Table<'s>, TableIter, std::slice::Iter<'r, TableRow<'s>>);
+children_iter!(
+    TableRow<'s>,
+    TableRowIter,
+    std::slice::Iter<'r, TableCell<'s>>
+);
+children_iter!(
+    VerseBlock<'s>,
+    VerseBlockIter,
+    std::slice::Iter<'r, Object<'s>>
+);
+empty_iter!(CommentBlock<'s>, CommentBlockIter);
+empty_iter!(ExampleBlock<'s>, ExampleBlockIter);
+empty_iter!(ExportBlock<'s>, ExportBlockIter);
+empty_iter!(SrcBlock<'s>, SrcBlockIter);
+empty_iter!(Clock<'s>, ClockIter);
+empty_iter!(DiarySexp<'s>, DiarySexpIter);
+empty_iter!(Planning<'s>, PlanningIter);
+empty_iter!(FixedWidthArea<'s>, FixedWidthAreaIter);
+empty_iter!(HorizontalRule<'s>, HorizontalRuleIter);
+empty_iter!(Keyword<'s>, KeywordIter);
+empty_iter!(BabelCall<'s>, BabelCallIter);
+empty_iter!(LatexEnvironment<'s>, LatexEnvironmentIter);
+children_iter!(Bold<'s>, BoldIter, std::slice::Iter<'r, Object<'s>>);
+children_iter!(Italic<'s>, ItalicIter, std::slice::Iter<'r, Object<'s>>);
+children_iter!(
+    Underline<'s>,
+    UnderlineIter,
+    std::slice::Iter<'r, Object<'s>>
+);
+children_iter!(
+    StrikeThrough<'s>,
+    StrikeThroughIter,
+    std::slice::Iter<'r, Object<'s>>
+);
+empty_iter!(Code<'s>, CodeIter);
+empty_iter!(Verbatim<'s>, VerbatimIter);
+empty_iter!(PlainText<'s>, PlainTextIter);
+empty_iter!(RegularLink<'s>, RegularLinkIter);
+children_iter!(
+    RadioLink<'s>,
+    RadioLinkIter,
+    std::slice::Iter<'r, Object<'s>>
+);
+children_iter!(
+    RadioTarget<'s>,
+    RadioTargetIter,
+    std::slice::Iter<'r, Object<'s>>
+);
+empty_iter!(PlainLink<'s>, PlainLinkIter);
+empty_iter!(AngleLink<'s>, AngleLinkIter);
+empty_iter!(OrgMacro<'s>, OrgMacroIter);
+empty_iter!(Entity<'s>, EntityIter);
+empty_iter!(LatexFragment<'s>, LatexFragmentIter);
+empty_iter!(ExportSnippet<'s>, ExportSnippetIter);
+multi_field_iter!(
+    FootnoteReference<'s>,
+    FootnoteReferenceIter,
+    definition,
+    std::slice::Iter<'r, Object<'s>>,
+);
+empty_iter!(Citation<'s>, CitationIter);
+empty_iter!(CitationReference<'s>, CitationReferenceIter);
+empty_iter!(InlineBabelCall<'s>, InlineBabelCallIter);
+empty_iter!(InlineSourceBlock<'s>, InlineSourceBlockIter);
+empty_iter!(LineBreak<'s>, LineBreakIter);
+empty_iter!(Target<'s>, TargetIter);
+empty_iter!(StatisticsCookie<'s>, StatisticsCookieIter);
+empty_iter!(Subscript<'s>, SubscriptIter);
+empty_iter!(Superscript<'s>, SuperscriptIter);
+children_iter!(
+    TableCell<'s>,
+    TableCellIter,
+    std::slice::Iter<'r, Object<'s>>
+);
+empty_iter!(Timestamp<'s>, TimestampIter);

src/iter/macros.rs

@@ -0,0 +1,114 @@
+/// Write the implementation of From<> to convert a borrow of the type to an AstNode
+macro_rules! to_ast_node {
+    ($inp:ty, $enum:expr) => {
+        impl<'r, 's> From<$inp> for AstNode<'r, 's> {
+            fn from(value: $inp) -> Self {
+                $enum(value)
+            }
+        }
+    };
+}
+
+pub(crate) use to_ast_node;
+
+/// Create iterators for ast nodes where it only has to iterate over children
+macro_rules! children_iter {
+    ($astnodetype:ty, $itertype:ident, $innertype:ty) => {
+        pub struct $itertype<'r, 's> {
+            next: $innertype,
+        }
+
+        impl<'r, 's> Iterator for $itertype<'r, 's> {
+            type Item = AstNode<'r, 's>;
+
+            fn next(&mut self) -> Option<Self::Item> {
+                self.next.next().map(Into::<AstNode>::into)
+            }
+        }
+
+        impl<'r, 's> IntoIterator for &'r $astnodetype {
+            type Item = AstNode<'r, 's>;
+
+            type IntoIter = $itertype<'r, 's>;
+
+            fn into_iter(self) -> Self::IntoIter {
+                $itertype {
+                    next: self.children.iter(),
+                }
+            }
+        }
+    };
+}
+
+pub(crate) use children_iter;
+
+/// Create iterators for ast nodes that do not contain any ast node children.
+macro_rules! empty_iter {
+    ($astnodetype:ty, $itertype:ident) => {
+        pub struct $itertype<'r, 's> {
+            phantom: PhantomData<&'r $astnodetype>,
+        }
+
+        impl<'r, 's> Iterator for $itertype<'r, 's> {
+            type Item = AstNode<'r, 's>;
+
+            fn next(&mut self) -> Option<Self::Item> {
+                None
+            }
+        }
+
+        impl<'r, 's> IntoIterator for &'r $astnodetype {
+            type Item = AstNode<'r, 's>;
+
+            type IntoIter = $itertype<'r, 's>;
+
+            fn into_iter(self) -> Self::IntoIter {
+                $itertype {
+                    phantom: PhantomData,
+                }
+            }
+        }
+    };
+}
+
+pub(crate) use empty_iter;
+
+/// Create iterators for ast nodes where it has to iterate over multiple child lists.
+macro_rules! multi_field_iter {
+    ($astnodetype:ty, $itertype:ident, $firstfieldname: ident, $firstinnertype:ty, $($fieldname: ident, $innertype:ty),*) => {
+        pub struct $itertype<'r, 's> {
+            $firstfieldname: $firstinnertype,
+            $(
+$fieldname: $innertype,
+),*
+        }
+
+        impl<'r, 's> Iterator for $itertype<'r, 's> {
+            type Item = AstNode<'r, 's>;
+
+            fn next(&mut self) -> Option<Self::Item> {
+                self.$firstfieldname.next().map(Into::<AstNode>::into)
+                $(
+.or_else(|| self.$fieldname.next().map(Into::<AstNode>::into))
+),*
+            }
+        }
+
+        impl<'r, 's> IntoIterator for &'r $astnodetype {
+            type Item = AstNode<'r, 's>;
+
+            type IntoIter = $itertype<'r, 's>;
+
+            fn into_iter(self) -> Self::IntoIter {
+                $itertype {
+            $firstfieldname: self.$firstfieldname.iter(),
+            $(
+$fieldname: self.$fieldname.iter(),
+),*
+                }
+            }
+        }
+    };
+}
+
+pub(crate) use multi_field_iter;

src/iter/mod.rs

@@ -0,0 +1,5 @@
+mod all_ast_node_iter;
+mod ast_node;
+mod ast_node_iter;
+mod macros;
+pub(crate) use ast_node::AstNode;

src/lib.rs

@@ -1,4 +1,3 @@
-#![feature(round_char_boundary)]
 #![feature(exit_status_error)]
 #![feature(trait_alias)]
 // TODO: #![warn(missing_docs)]
@@ -8,6 +7,7 @@ pub mod compare;
 
 mod context;
 mod error;
+mod iter;
 pub mod parser;
 pub mod types;
 

src/parser/citation.rs

@@ -187,7 +187,7 @@ mod tests {
     use crate::context::List;
     use crate::parser::element_parser::element;
     use crate::types::Element;
-    use crate::types::Source;
+    use crate::types::GetStandardProperties;
 
     #[test]
     fn citation_simple() {
@@ -202,7 +202,10 @@ mod tests {
             _ => panic!("Should be a paragraph!"),
         };
         assert_eq!(Into::<&str>::into(remaining), "");
-        assert_eq!(first_paragraph.get_source(), "[cite:@foo]");
+        assert_eq!(
+            first_paragraph.get_standard_properties().get_source(),
+            "[cite:@foo]"
+        );
         assert_eq!(first_paragraph.children.len(), 1);
         assert_eq!(
             first_paragraph

src/parser/document.rs

@@ -7,8 +7,6 @@ use super::in_buffer_settings::apply_in_buffer_settings;
 use super::in_buffer_settings::scan_for_in_buffer_settings;
 use super::org_source::OrgSource;
 use super::section::zeroth_section;
-use super::token::AllTokensIterator;
-use super::token::Token;
 use super::util::get_consumed;
 use crate::context::parser_with_context;
 use crate::context::Context;
@@ -19,6 +17,7 @@ use crate::context::RefContext;
 use crate::error::CustomError;
 use crate::error::MyError;
 use crate::error::Res;
+use crate::iter::AstNode;
 use crate::parser::org_source::convert_error;
 use crate::parser::util::blank_line;
 use crate::types::Document;
@@ -111,15 +110,14 @@ fn document_org_source<'b, 'g, 'r, 's>(
         _document(context, input).map(|(rem, out)| (Into::<&str>::into(rem), out))?;
     {
         // If there are radio targets in this document then we need to parse the entire document again with the knowledge of the radio targets.
-        let all_radio_targets: Vec<&Vec<Object<'_>>> = document
-            .iter_tokens()
-            .filter_map(|tkn| match tkn {
-                Token::Object(obj) => Some(obj),
-                _ => None,
-            })
-            .filter_map(|obj| match obj {
-                Object::RadioTarget(rt) => Some(rt),
-                _ => None,
+        let all_radio_targets: Vec<&Vec<Object<'_>>> = Into::<AstNode>::into(&document)
+            .into_iter()
+            .filter_map(|ast_node| {
+                if let AstNode::RadioTarget(ast_node) = ast_node {
+                    Some(ast_node)
+                } else {
+                    None
+                }
             })
             .map(|rt| &rt.children)
             .collect();
@@ -155,9 +153,3 @@ fn _document<'b, 'g, 'r, 's>(
         },
     ))
 }
-
-impl<'s> Document<'s> {
-    fn iter_tokens<'r>(&'r self) -> impl Iterator<Item = Token<'r, 's>> {
-        AllTokensIterator::new(Token::Document(self))
-    }
-}

src/parser/dynamic_block.rs

@@ -40,7 +40,6 @@ pub(crate) fn dynamic_block<'b, 'g, 'r, 's>(
     context: RefContext<'b, 'g, 'r, 's>,
     input: OrgSource<'s>,
 ) -> Res<OrgSource<'s>, DynamicBlock<'s>> {
-    // TODO: Do I need to differentiate between different dynamic block types.
     if immediate_in_section(context, "dynamic block") {
         return Err(nom::Err::Error(CustomError::MyError(MyError(
             "Cannot nest objects of the same element".into(),

src/parser/element_parser.rs

@@ -107,6 +107,7 @@ fn _element<'b, 'g, 'r, 's>(
                 match map(paragraph_matcher, Element::Paragraph)(remaining) {
                     the_ok @ Ok(_) => the_ok,
                     Err(_) => {
+                        // TODO: Because this function expects a single element, if there are multiple affiliated keywords before an element that cannot have affiliated keywords, we end up re-parsing the affiliated keywords many times.
                         affiliated_keywords.clear();
                         map(affiliated_keyword_matcher, Element::Keyword)(input)
                     }

src/parser/footnote_definition.rs

@@ -129,7 +129,7 @@ mod tests {
     use crate::context::Context;
     use crate::context::GlobalSettings;
     use crate::context::List;
-    use crate::types::Source;
+    use crate::types::GetStandardProperties;
 
     #[test]
     fn two_paragraphs() {
@@ -150,13 +150,17 @@ line footnote.",
             footnote_definition_matcher(remaining).expect("Parse second footnote_definition.");
         assert_eq!(Into::<&str>::into(remaining), "");
         assert_eq!(
-            first_footnote_definition.get_source(),
+            first_footnote_definition
+                .get_standard_properties()
+                .get_source(),
             "[fn:1] A footnote.
 
 "
         );
         assert_eq!(
-            second_footnote_definition.get_source(),
+            second_footnote_definition
+                .get_standard_properties()
+                .get_source(),
             "[fn:2] A multi-
 
 line footnote."
@@ -181,7 +185,9 @@ not in the footnote.",
             footnote_definition_matcher(input).expect("Parse first footnote_definition");
         assert_eq!(Into::<&str>::into(remaining), "not in the footnote.");
         assert_eq!(
-            first_footnote_definition.get_source(),
+            first_footnote_definition
+                .get_standard_properties()
+                .get_source(),
             "[fn:2] A multi-
 
 line footnote.

src/parser/in_buffer_settings.rs

@@ -1,17 +1,15 @@
 use nom::branch::alt;
 use nom::bytes::complete::is_not;
 use nom::bytes::complete::tag_no_case;
-use nom::character::complete::anychar;
+use nom::bytes::complete::take_until;
 use nom::character::complete::space1;
-use nom::combinator::map;
-use nom::multi::many0;
-use nom::multi::many_till;
 use nom::multi::separated_list0;
 
 use super::keyword::filtered_keyword;
 use super::keyword_todo::todo_keywords;
 use super::OrgSource;
 use crate::context::HeadlineLevelFilter;
+use crate::error::CustomError;
 use crate::error::Res;
 use crate::types::Keyword;
 use crate::GlobalSettings;
@@ -20,13 +18,40 @@ use crate::GlobalSettings;
 pub(crate) fn scan_for_in_buffer_settings<'s>(
     input: OrgSource<'s>,
 ) -> Res<OrgSource<'s>, Vec<Keyword<'s>>> {
-    // TODO: Optimization idea: since this is slicing the OrgSource at each character, it might be more efficient to do a parser that uses a search function like take_until, and wrap it in a function similar to consumed but returning the input along with the normal output, then pass all of that into a verify that confirms we were at the start of a line using the input we just returned.
+    // TODO: Write some tests to make sure this is functioning properly.
 
-    let keywords = many0(map(
-        many_till(anychar, filtered_keyword(in_buffer_settings_key)),
-        |(_, kw)| kw,
-    ))(input);
-    keywords
+    let mut keywords = Vec::new();
+    let mut remaining = input;
+    loop {
+        // Skip text until possible in_buffer_setting
+        let start_of_pound = take_until::<_, _, CustomError<_>>("#+")(remaining);
+        let start_of_pound = if let Ok((start_of_pound, _)) = start_of_pound {
+            start_of_pound
+        } else {
+            break;
+        };
+        // Go backwards to the start of the line and run the filtered_keyword parser
+        let start_of_line = start_of_pound.get_start_of_line();
+
+        let (remain, maybe_kw) = match filtered_keyword(in_buffer_settings_key)(start_of_line) {
+            Ok((remain, kw)) => (remain, Some(kw)),
+            Err(_) => {
+                let end_of_line = take_until::<_, _, CustomError<_>>("\n")(start_of_pound);
+                if let Ok((end_of_line, _)) = end_of_line {
+                    (end_of_line, None)
+                } else {
+                    break;
+                }
+            }
+        };
+
+        if let Some(kw) = maybe_kw {
+            keywords.push(kw);
+        }
+        remaining = remain;
+    }
+
+    Ok((remaining, keywords))
 }
 
 #[cfg_attr(feature = "tracing", tracing::instrument(ret, level = "debug"))]
@@ -88,3 +113,33 @@ pub(crate) fn apply_in_buffer_settings<'g, 's, 'sf>(
 
     Ok(new_settings)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn scan_test() -> Result<(), Box<dyn std::error::Error>> {
+        let input = OrgSource::new(
+            r#"
+foo
+   #+archive: bar
+
+baz #+category: lorem
+
+#+label: ipsum
+
+#+todo: dolar
+cat
+"#,
+        );
+        let (remaining, settings) = scan_for_in_buffer_settings(input)?;
+        assert_eq!(Into::<&str>::into(remaining), "cat\n");
+        let keys: Vec<_> = settings.iter().map(|kw| kw.key).collect();
+        // category is skipped because it is not the first non-whitespace on the line.
+        //
+        // label is skipped because it is not an in-buffer setting.
+        assert_eq!(keys, vec!["archive", "todo"]);
+        Ok(())
+    }
+}

src/parser/keyword.rs

@@ -24,6 +24,7 @@ use crate::error::CustomError;
 use crate::error::MyError;
 use crate::error::Res;
 use crate::parser::util::start_of_line;
+use crate::types::BabelCall;
 use crate::types::Keyword;
 
 const ORG_ELEMENT_AFFILIATED_KEYWORDS: [&'static str; 13] = [
@@ -103,8 +104,16 @@ pub(crate) fn affiliated_keyword<'b, 'g, 'r, 's>(
 pub(crate) fn babel_call_keyword<'b, 'g, 'r, 's>(
     _context: RefContext<'b, 'g, 'r, 's>,
     input: OrgSource<'s>,
-) -> Res<OrgSource<'s>, Keyword<'s>> {
-    filtered_keyword(babel_call_key)(input)
+) -> Res<OrgSource<'s>, BabelCall<'s>> {
+    let (remaining, kw) = filtered_keyword(babel_call_key)(input)?;
+    Ok((
+        remaining,
+        BabelCall {
+            source: kw.source,
+            key: kw.key,
+            value: kw.value,
+        },
+    ))
 }
 
 #[cfg_attr(feature = "tracing", tracing::instrument(ret, level = "debug"))]

src/parser/lesser_block.rs

@@ -261,11 +261,10 @@ fn data<'s>(input: OrgSource<'s>) -> Res<OrgSource<'s>, OrgSource<'s>> {
     is_not("\r\n")(input)
 }
 
-fn lesser_block_end(current_name: &str) -> impl ContextMatcher {
-    let current_name_lower = current_name.to_lowercase();
-    move |context, input: OrgSource<'_>| {
-        _lesser_block_end(context, input, current_name_lower.as_str())
-    }
+fn lesser_block_end<'c>(current_name: &'c str) -> impl ContextMatcher + 'c {
+    // Since the lesser block names are statically defined in code, we can simply assert that the name is lowercase instead of causing an allocation by converting to lowercase.
+    debug_assert!(current_name == current_name.to_lowercase());
+    move |context, input: OrgSource<'_>| _lesser_block_end(context, input, current_name)
 }
 
 #[cfg_attr(feature = "tracing", tracing::instrument(ret, level = "debug"))]

src/parser/mod.rs

@@ -43,7 +43,6 @@ mod table;
 mod target;
 mod text_markup;
 mod timestamp;
-mod token;
 mod util;
 pub use document::parse;
 pub use document::parse_with_settings;

src/parser/org_source.rs

@@ -1,6 +1,7 @@
 use std::ops::RangeBounds;
 
 use nom::Compare;
+use nom::FindSubstring;
 use nom::InputIter;
 use nom::InputLength;
 use nom::InputTake;
@@ -72,11 +73,60 @@ impl<'s> OrgSource<'s> {
     }
 
     pub(crate) fn get_until(&self, other: OrgSource<'s>) -> OrgSource<'s> {
-        assert!(other.start >= self.start);
-        assert!(other.end <= self.end);
+        debug_assert!(other.start >= self.start);
+        debug_assert!(other.end <= self.end);
         self.slice(..(other.start - self.start))
     }
 
+    pub(crate) fn get_start_of_line(&self) -> OrgSource<'s> {
+        let skipped_text = self.text_since_line_break();
+        let mut bracket_depth = self.bracket_depth;
+        let mut brace_depth = self.brace_depth;
+        let mut parenthesis_depth = self.parenthesis_depth;
+        // Since we're going backwards, this does the opposite.
+        for byte in skipped_text.bytes() {
+            match byte {
+                b'\n' => {
+                    panic!("Should not hit a line break when only going back to the start of the line.");
+                }
+                b'[' => {
+                    bracket_depth -= 1;
+                }
+                b']' => {
+                    bracket_depth += 1;
+                }
+                b'{' => {
+                    brace_depth -= 1;
+                }
+                b'}' => {
+                    brace_depth += 1;
+                }
+                b'(' => {
+                    parenthesis_depth -= 1;
+                }
+                b')' => {
+                    parenthesis_depth += 1;
+                }
+                _ => {}
+            };
+        }
+
+        OrgSource {
+            full_source: self.full_source,
+            start: self.start_of_line,
+            end: self.end,
+            start_of_line: self.start_of_line,
+            preceding_character: if self.start_of_line > 0 {
+                Some('\n')
+            } else {
+                None
+            },
+            bracket_depth,
+            brace_depth,
+            parenthesis_depth,
+        }
+    }
+
     pub(crate) fn get_bracket_depth(&self) -> BracketDepth {
         self.bracket_depth
     }
@@ -310,6 +360,12 @@ impl<'s> InputTakeAtPosition for OrgSource<'s> {
     }
 }
 
+impl<'n, 's> FindSubstring<&'n str> for OrgSource<'s> {
+    fn find_substring(&self, substr: &'n str) -> Option<usize> {
+        Into::<&str>::into(self).find(substr)
+    }
+}
+
 pub(crate) fn convert_error<'a, I: Into<CustomError<&'a str>>>(
     err: nom::Err<I>,
 ) -> nom::Err<CustomError<&'a str>> {

src/parser/paragraph.rs

@@ -74,7 +74,7 @@ mod tests {
     use crate::context::List;
     use crate::parser::element_parser::element;
     use crate::parser::org_source::OrgSource;
-    use crate::types::Source;
+    use crate::types::GetStandardProperties;
 
     #[test]
     fn two_paragraphs() {
@@ -87,7 +87,13 @@ mod tests {
         let (remaining, second_paragraph) =
             paragraph_matcher(remaining).expect("Parse second paragraph.");
         assert_eq!(Into::<&str>::into(remaining), "");
-        assert_eq!(first_paragraph.get_source(), "foo bar baz\n\n");
-        assert_eq!(second_paragraph.get_source(), "lorem ipsum");
+        assert_eq!(
+            first_paragraph.get_standard_properties().get_source(),
+            "foo bar baz\n\n"
+        );
+        assert_eq!(
+            second_paragraph.get_standard_properties().get_source(),
+            "lorem ipsum"
+        );
     }
 }

src/parser/plain_list.rs

@@ -445,7 +445,7 @@ mod tests {
     use crate::context::Context;
     use crate::context::GlobalSettings;
     use crate::context::List;
-    use crate::types::Source;
+    use crate::types::GetStandardProperties;
 
     #[test]
     fn plain_list_item_empty() {
@@ -456,7 +456,7 @@ mod tests {
         let plain_list_item_matcher = parser_with_context!(plain_list_item)(&initial_context);
         let (remaining, result) = plain_list_item_matcher(input).unwrap();
         assert_eq!(Into::<&str>::into(remaining), "");
-        assert_eq!(result.source, "1.");
+        assert_eq!(result.get_standard_properties().get_source(), "1.");
     }
 
     #[test]
@@ -468,7 +468,7 @@ mod tests {
         let plain_list_item_matcher = parser_with_context!(plain_list_item)(&initial_context);
         let (remaining, result) = plain_list_item_matcher(input).unwrap();
         assert_eq!(Into::<&str>::into(remaining), "");
-        assert_eq!(result.source, "1. foo");
+        assert_eq!(result.get_standard_properties().get_source(), "1. foo");
     }
 
     #[test]
@@ -480,7 +480,7 @@ mod tests {
         let plain_list_matcher = parser_with_context!(plain_list)(&initial_context);
         let (remaining, result) = plain_list_matcher(input).unwrap();
         assert_eq!(Into::<&str>::into(remaining), "");
-        assert_eq!(result.source, "1.");
+        assert_eq!(result.get_standard_properties().get_source(), "1.");
     }
 
     #[test]
@@ -492,7 +492,7 @@ mod tests {
         let plain_list_matcher = parser_with_context!(plain_list)(&initial_context);
         let (remaining, result) = plain_list_matcher(input).unwrap();
         assert_eq!(Into::<&str>::into(remaining), "");
-        assert_eq!(result.source, "1. foo");
+        assert_eq!(result.get_standard_properties().get_source(), "1. foo");
     }
 
     #[test]
@@ -539,7 +539,7 @@ mod tests {
             plain_list_matcher(input).expect("Should parse the plain list successfully.");
         assert_eq!(Into::<&str>::into(remaining), "   ipsum\n");
         assert_eq!(
-            result.get_source(),
+            result.get_standard_properties().get_source(),
             r#"1. foo
 2. bar
    baz
@@ -567,7 +567,7 @@ baz"#,
             plain_list_matcher(input).expect("Should parse the plain list successfully.");
         assert_eq!(Into::<&str>::into(remaining), "baz");
         assert_eq!(
-            result.get_source(),
+            result.get_standard_properties().get_source(),
             r#"1. foo
    1. bar
 
@@ -600,7 +600,7 @@ dolar"#,
             plain_list_matcher(input).expect("Should parse the plain list successfully.");
         assert_eq!(Into::<&str>::into(remaining), "dolar");
         assert_eq!(
-            result.get_source(),
+            result.get_standard_properties().get_source(),
             r#"1. foo
 
    bar

src/parser/plain_text.rs

@@ -146,7 +146,7 @@ mod tests {
     use crate::context::GlobalSettings;
     use crate::context::List;
     use crate::parser::object_parser::detect_standard_set_object_sans_plain_text;
-    use crate::types::Source;
+    use crate::types::GetStandardProperties;
 
     #[test]
     fn plain_text_simple() {
@@ -159,6 +159,9 @@ mod tests {
         ))(&initial_context);
         let (remaining, result) = map(plain_text_matcher, Object::PlainText)(input).unwrap();
         assert_eq!(Into::<&str>::into(remaining), "");
-        assert_eq!(result.get_source(), Into::<&str>::into(input));
+        assert_eq!(
+            result.get_standard_properties().get_source(),
+            Into::<&str>::into(input)
+        );
     }
 }

src/parser/radio_link.rs

@@ -151,8 +151,8 @@ mod tests {
     use crate::parser::element_parser::element;
     use crate::types::Bold;
     use crate::types::Element;
+    use crate::types::GetStandardProperties;
     use crate::types::PlainText;
-    use crate::types::Source;
 
     #[test]
     fn plain_text_radio_target() {
@@ -172,7 +172,10 @@ mod tests {
             _ => panic!("Should be a paragraph!"),
         };
         assert_eq!(Into::<&str>::into(remaining), "");
-        assert_eq!(first_paragraph.get_source(), "foo bar baz");
+        assert_eq!(
+            first_paragraph.get_standard_properties().get_source(),
+            "foo bar baz"
+        );
         assert_eq!(first_paragraph.children.len(), 3);
         assert_eq!(
             first_paragraph
@@ -208,7 +211,10 @@ mod tests {
             _ => panic!("Should be a paragraph!"),
         };
         assert_eq!(Into::<&str>::into(remaining), "");
-        assert_eq!(first_paragraph.get_source(), "foo *bar* baz");
+        assert_eq!(
+            first_paragraph.get_standard_properties().get_source(),
+            "foo *bar* baz"
+        );
         assert_eq!(first_paragraph.children.len(), 3);
         assert_eq!(
             first_paragraph

src/parser/token.rs

@@ -1,135 +0,0 @@
-use std::collections::VecDeque;
-
-use crate::types::Document;
-use crate::types::DocumentElement;
-use crate::types::Element;
-use crate::types::Heading;
-use crate::types::NodeProperty;
-use crate::types::Object;
-use crate::types::PlainListItem;
-use crate::types::Section;
-use crate::types::TableCell;
-use crate::types::TableRow;
-
-pub(crate) enum Token<'r, 's> {
-    Document(&'r Document<'s>),
-    Heading(&'r Heading<'s>),
-    Section(&'r Section<'s>),
-    Object(&'r Object<'s>),
-    Element(&'r Element<'s>),
-    PlainListItem(&'r PlainListItem<'s>),
-    TableRow(&'r TableRow<'s>),
-    TableCell(&'r TableCell<'s>),
-    NodeProperty(&'r NodeProperty<'s>),
-}
-
-impl<'r, 's> Token<'r, 's> {
-    fn iter_tokens(&self) -> Box<dyn Iterator<Item = Token<'r, 's>> + '_> {
-        match self {
-            Token::Document(document) => Box::new(
-                document
-                    .zeroth_section
-                    .iter()
-                    .map(Token::Section)
-                    .chain(document.children.iter().map(Token::Heading)),
-            ),
-            Token::Heading(heading) => Box::new(heading.title.iter().map(Token::Object).chain(
-                heading.children.iter().map(|de| match de {
-                    DocumentElement::Heading(ref obj) => Token::Heading(obj),
-                    DocumentElement::Section(ref obj) => Token::Section(obj),
-                }),
-            )),
-            Token::Section(section) => Box::new(section.children.iter().map(Token::Element)),
-            Token::Object(obj) => match obj {
-                Object::Bold(inner) => Box::new(inner.children.iter().map(Token::Object)),
-                Object::Italic(inner) => Box::new(inner.children.iter().map(Token::Object)),
-                Object::Underline(inner) => Box::new(inner.children.iter().map(Token::Object)),
-                Object::StrikeThrough(inner) => Box::new(inner.children.iter().map(Token::Object)),
-                Object::Code(_) => Box::new(std::iter::empty()),
-                Object::Verbatim(_) => Box::new(std::iter::empty()),
-                Object::PlainText(_) => Box::new(std::iter::empty()),
-                Object::RegularLink(_) => Box::new(std::iter::empty()),
-                Object::RadioLink(inner) => Box::new(inner.children.iter().map(Token::Object)),
-                Object::RadioTarget(inner) => Box::new(inner.children.iter().map(Token::Object)),
-                Object::PlainLink(_) => Box::new(std::iter::empty()),
-                Object::AngleLink(_) => Box::new(std::iter::empty()),
-                Object::OrgMacro(_) => Box::new(std::iter::empty()),
-                Object::Entity(_) => Box::new(std::iter::empty()),
-                Object::LatexFragment(_) => Box::new(std::iter::empty()),
-                Object::ExportSnippet(_) => Box::new(std::iter::empty()),
-                Object::FootnoteReference(inner) => {
-                    Box::new(inner.definition.iter().map(Token::Object))
-                }
-                Object::Citation(_) => Box::new(std::iter::empty()), // TODO: Iterate over children
-                Object::CitationReference(_) => Box::new(std::iter::empty()), // TODO: Iterate over children
-                Object::InlineBabelCall(_) => Box::new(std::iter::empty()),
-                Object::InlineSourceBlock(_) => Box::new(std::iter::empty()),
-                Object::LineBreak(_) => Box::new(std::iter::empty()),
-                Object::Target(_) => Box::new(std::iter::empty()),
-                Object::StatisticsCookie(_) => Box::new(std::iter::empty()),
-                Object::Subscript(_) => Box::new(std::iter::empty()), // TODO: Iterate over children
-                Object::Superscript(_) => Box::new(std::iter::empty()), // TODO: Iterate over children
-                Object::Timestamp(_) => Box::new(std::iter::empty()),
-            },
-            Token::Element(elem) => match elem {
-                Element::Paragraph(inner) => Box::new(inner.children.iter().map(Token::Object)),
-                Element::PlainList(inner) => {
-                    Box::new(inner.children.iter().map(Token::PlainListItem))
-                }
-                Element::GreaterBlock(inner) => Box::new(inner.children.iter().map(Token::Element)),
-                Element::DynamicBlock(inner) => Box::new(inner.children.iter().map(Token::Element)),
-                Element::FootnoteDefinition(inner) => {
-                    Box::new(inner.children.iter().map(Token::Element))
-                }
-                Element::Comment(_) => Box::new(std::iter::empty()),
-                Element::Drawer(inner) => Box::new(inner.children.iter().map(Token::Element)),
-                Element::PropertyDrawer(inner) => {
-                    Box::new(inner.children.iter().map(Token::NodeProperty))
-                }
-                Element::Table(inner) => Box::new(inner.children.iter().map(Token::TableRow)),
-                Element::VerseBlock(inner) => Box::new(inner.children.iter().map(Token::Object)),
-                Element::CommentBlock(_) => Box::new(std::iter::empty()),
-                Element::ExampleBlock(_) => Box::new(std::iter::empty()),
-                Element::ExportBlock(_) => Box::new(std::iter::empty()),
-                Element::SrcBlock(_) => Box::new(std::iter::empty()),
-                Element::Clock(_) => Box::new(std::iter::empty()),
-                Element::DiarySexp(_) => Box::new(std::iter::empty()),
-                Element::Planning(_) => Box::new(std::iter::empty()),
-                Element::FixedWidthArea(_) => Box::new(std::iter::empty()),
-                Element::HorizontalRule(_) => Box::new(std::iter::empty()),
-                Element::Keyword(_) => Box::new(std::iter::empty()),
-                Element::BabelCall(_) => Box::new(std::iter::empty()),
-                Element::LatexEnvironment(_) => Box::new(std::iter::empty()),
-            },
-            Token::PlainListItem(elem) => Box::new(elem.children.iter().map(Token::Element)),
-            Token::TableRow(elem) => Box::new(elem.children.iter().map(Token::TableCell)),
-            Token::TableCell(elem) => Box::new(elem.children.iter().map(Token::Object)),
-            Token::NodeProperty(_) => Box::new(std::iter::empty()),
-        }
-    }
-}
-
-pub(crate) struct AllTokensIterator<'r, 's> {
-    queued_tokens: VecDeque<Token<'r, 's>>,
-}
-
-impl<'r, 's> AllTokensIterator<'r, 's> {
-    pub(crate) fn new(tkn: Token<'r, 's>) -> Self {
-        let mut queued_tokens = VecDeque::new();
-        queued_tokens.push_back(tkn);
-        AllTokensIterator { queued_tokens }
-    }
-}
-
-impl<'r, 's> Iterator for AllTokensIterator<'r, 's> {
-    type Item = Token<'r, 's>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        let next_token = match self.queued_tokens.pop_front() {
-            Some(tkn) => tkn,
-            None => return None,
-        };
-        self.queued_tokens.extend(next_token.iter_tokens());
-        Some(next_token)
-    }
-}

src/types/document.rs

@@ -1,6 +1,7 @@
 use super::Element;
+use super::GetStandardProperties;
 use super::Object;
-use super::Source;
+use super::StandardProperties;
 
 pub type PriorityCookie = u8;
 pub type HeadlineLevel = u16;
@@ -43,28 +44,28 @@ pub enum TodoKeywordType {
     Done,
 }
 
-impl<'s> Source<'s> for Document<'s> {
-    fn get_source(&'s self) -> &'s str {
-        self.source
-    }
-}
-
-impl<'s> Source<'s> for DocumentElement<'s> {
-    fn get_source(&'s self) -> &'s str {
+impl<'s> GetStandardProperties<'s> for DocumentElement<'s> {
+    fn get_standard_properties(&'s self) -> &'s dyn StandardProperties {
         match self {
-            DocumentElement::Heading(obj) => obj.source,
-            DocumentElement::Section(obj) => obj.source,
+            DocumentElement::Heading(inner) => inner,
+            DocumentElement::Section(inner) => inner,
         }
     }
 }
 
-impl<'s> Source<'s> for Section<'s> {
+impl<'s> StandardProperties<'s> for Document<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Heading<'s> {
+impl<'s> StandardProperties<'s> for Section<'s> {
+    fn get_source(&'s self) -> &'s str {
+        self.source
+    }
+}
+
+impl<'s> StandardProperties<'s> for Heading<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }

src/types/element.rs

@@ -4,6 +4,7 @@ use super::greater_element::GreaterBlock;
 use super::greater_element::PlainList;
 use super::greater_element::PropertyDrawer;
 use super::greater_element::Table;
+use super::lesser_element::BabelCall;
 use super::lesser_element::Clock;
 use super::lesser_element::Comment;
 use super::lesser_element::CommentBlock;
@@ -19,8 +20,9 @@ use super::lesser_element::Planning;
 use super::lesser_element::SrcBlock;
 use super::lesser_element::VerseBlock;
 use super::Drawer;
+use super::GetStandardProperties;
 use super::SetSource;
-use super::Source;
+use super::StandardProperties;
 
 #[derive(Debug)]
 pub enum Element<'s> {
@@ -44,39 +46,10 @@ pub enum Element<'s> {
     FixedWidthArea(FixedWidthArea<'s>),
     HorizontalRule(HorizontalRule<'s>),
     Keyword(Keyword<'s>),
-    BabelCall(Keyword<'s>),
+    BabelCall(BabelCall<'s>),
     LatexEnvironment(LatexEnvironment<'s>),
 }
 
-impl<'s> Source<'s> for Element<'s> {
-    fn get_source(&'s self) -> &'s str {
-        match self {
-            Element::Paragraph(obj) => obj.get_source(),
-            Element::PlainList(obj) => obj.get_source(),
-            Element::GreaterBlock(obj) => obj.get_source(),
-            Element::DynamicBlock(obj) => obj.get_source(),
-            Element::FootnoteDefinition(obj) => obj.get_source(),
-            Element::Comment(obj) => obj.get_source(),
-            Element::Drawer(obj) => obj.get_source(),
-            Element::PropertyDrawer(obj) => obj.get_source(),
-            Element::Table(obj) => obj.get_source(),
-            Element::VerseBlock(obj) => obj.get_source(),
-            Element::CommentBlock(obj) => obj.get_source(),
-            Element::ExampleBlock(obj) => obj.get_source(),
-            Element::ExportBlock(obj) => obj.get_source(),
-            Element::SrcBlock(obj) => obj.get_source(),
-            Element::Clock(obj) => obj.get_source(),
-            Element::DiarySexp(obj) => obj.get_source(),
-            Element::Planning(obj) => obj.get_source(),
-            Element::FixedWidthArea(obj) => obj.get_source(),
-            Element::HorizontalRule(obj) => obj.get_source(),
-            Element::Keyword(obj) => obj.get_source(),
-            Element::BabelCall(obj) => obj.get_source(),
-            Element::LatexEnvironment(obj) => obj.get_source(),
-        }
-    }
-}
-
 impl<'s> SetSource<'s> for Element<'s> {
     #[cfg_attr(feature = "tracing", tracing::instrument(ret, level = "debug"))]
     fn set_source(&mut self, source: &'s str) {
@@ -106,3 +79,32 @@ impl<'s> SetSource<'s> for Element<'s> {
         }
     }
 }
+
+impl<'s> GetStandardProperties<'s> for Element<'s> {
+    fn get_standard_properties(&'s self) -> &'s dyn StandardProperties {
+        match self {
+            Element::Paragraph(inner) => inner,
+            Element::PlainList(inner) => inner,
+            Element::GreaterBlock(inner) => inner,
+            Element::DynamicBlock(inner) => inner,
+            Element::FootnoteDefinition(inner) => inner,
+            Element::Comment(inner) => inner,
+            Element::Drawer(inner) => inner,
+            Element::PropertyDrawer(inner) => inner,
+            Element::Table(inner) => inner,
+            Element::VerseBlock(inner) => inner,
+            Element::CommentBlock(inner) => inner,
+            Element::ExampleBlock(inner) => inner,
+            Element::ExportBlock(inner) => inner,
+            Element::SrcBlock(inner) => inner,
+            Element::Clock(inner) => inner,
+            Element::DiarySexp(inner) => inner,
+            Element::Planning(inner) => inner,
+            Element::FixedWidthArea(inner) => inner,
+            Element::HorizontalRule(inner) => inner,
+            Element::Keyword(inner) => inner,
+            Element::BabelCall(inner) => inner,
+            Element::LatexEnvironment(inner) => inner,
+        }
+    }
+}

src/types/get_standard_properties.rs

@@ -0,0 +1,12 @@
+use super::StandardProperties;
+
+pub trait GetStandardProperties<'s> {
+    // TODO: Can I eliminate this dynamic dispatch, perhaps using nominal generic structs? Low prioritiy since this is not used during parsing.
+    fn get_standard_properties(&'s self) -> &'s dyn StandardProperties;
+}
+
+impl<'s, I: StandardProperties<'s>> GetStandardProperties<'s> for I {
+    fn get_standard_properties(&'s self) -> &'s dyn StandardProperties {
+        self
+    }
+}

src/types/greater_element.rs

@@ -2,7 +2,7 @@ use super::element::Element;
 use super::lesser_element::TableCell;
 use super::Keyword;
 use super::Object;
-use super::Source;
+use super::StandardProperties;
 
 #[derive(Debug)]
 pub struct PlainList<'s> {
@@ -85,61 +85,61 @@ pub struct TableRow<'s> {
     pub children: Vec<TableCell<'s>>,
 }
 
-impl<'s> Source<'s> for PlainList<'s> {
+impl<'s> StandardProperties<'s> for PlainList<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for PlainListItem<'s> {
+impl<'s> StandardProperties<'s> for PlainListItem<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for GreaterBlock<'s> {
+impl<'s> StandardProperties<'s> for GreaterBlock<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for DynamicBlock<'s> {
+impl<'s> StandardProperties<'s> for DynamicBlock<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for FootnoteDefinition<'s> {
+impl<'s> StandardProperties<'s> for FootnoteDefinition<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Drawer<'s> {
+impl<'s> StandardProperties<'s> for Drawer<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for PropertyDrawer<'s> {
+impl<'s> StandardProperties<'s> for PropertyDrawer<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for NodeProperty<'s> {
+impl<'s> StandardProperties<'s> for NodeProperty<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Table<'s> {
+impl<'s> StandardProperties<'s> for Table<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for TableRow<'s> {
+impl<'s> StandardProperties<'s> for TableRow<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }

src/types/lesser_element.rs

@@ -1,6 +1,6 @@
 use super::object::Object;
 use super::PlainText;
-use super::Source;
+use super::StandardProperties;
 
 #[derive(Debug)]
 pub struct Paragraph<'s> {
@@ -91,6 +91,13 @@ pub struct Keyword<'s> {
     pub value: &'s str,
 }
 
+#[derive(Debug)]
+pub struct BabelCall<'s> {
+    pub source: &'s str,
+    pub key: &'s str,
+    pub value: &'s str,
+}
+
 #[derive(Debug)]
 pub struct LatexEnvironment<'s> {
     pub source: &'s str,
@@ -107,87 +114,93 @@ impl<'s> Paragraph<'s> {
     }
 }
 
-impl<'s> Source<'s> for Paragraph<'s> {
+impl<'s> StandardProperties<'s> for Paragraph<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for TableCell<'s> {
+impl<'s> StandardProperties<'s> for TableCell<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Comment<'s> {
+impl<'s> StandardProperties<'s> for Comment<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for VerseBlock<'s> {
+impl<'s> StandardProperties<'s> for VerseBlock<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
-impl<'s> Source<'s> for CommentBlock<'s> {
+impl<'s> StandardProperties<'s> for CommentBlock<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
-impl<'s> Source<'s> for ExampleBlock<'s> {
+impl<'s> StandardProperties<'s> for ExampleBlock<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
-impl<'s> Source<'s> for ExportBlock<'s> {
+impl<'s> StandardProperties<'s> for ExportBlock<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
-impl<'s> Source<'s> for SrcBlock<'s> {
+impl<'s> StandardProperties<'s> for SrcBlock<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Clock<'s> {
+impl<'s> StandardProperties<'s> for Clock<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for DiarySexp<'s> {
+impl<'s> StandardProperties<'s> for DiarySexp<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Planning<'s> {
+impl<'s> StandardProperties<'s> for Planning<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for FixedWidthArea<'s> {
+impl<'s> StandardProperties<'s> for FixedWidthArea<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for HorizontalRule<'s> {
+impl<'s> StandardProperties<'s> for HorizontalRule<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Keyword<'s> {
+impl<'s> StandardProperties<'s> for Keyword<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for LatexEnvironment<'s> {
+impl<'s> StandardProperties<'s> for BabelCall<'s> {
+    fn get_source(&'s self) -> &'s str {
+        self.source
+    }
+}
+
+impl<'s> StandardProperties<'s> for LatexEnvironment<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }

src/types/mod.rs

@@ -1,9 +1,11 @@
 mod document;
 mod element;
+mod get_standard_properties;
 mod greater_element;
 mod lesser_element;
 mod object;
 mod source;
+mod standard_properties;
 pub use document::Document;
 pub use document::DocumentElement;
 pub use document::Heading;
@@ -12,6 +14,7 @@ pub use document::PriorityCookie;
 pub use document::Section;
 pub use document::TodoKeywordType;
 pub use element::Element;
+pub use get_standard_properties::GetStandardProperties;
 pub use greater_element::CheckboxType;
 pub use greater_element::Drawer;
 pub use greater_element::DynamicBlock;
@@ -24,6 +27,7 @@ pub use greater_element::PlainListItem;
 pub use greater_element::PropertyDrawer;
 pub use greater_element::Table;
 pub use greater_element::TableRow;
+pub use lesser_element::BabelCall;
 pub use lesser_element::Clock;
 pub use lesser_element::Comment;
 pub use lesser_element::CommentBlock;
@@ -68,4 +72,4 @@ pub use object::Timestamp;
 pub use object::Underline;
 pub use object::Verbatim;
 pub(crate) use source::SetSource;
-pub use source::Source;
+pub use standard_properties::StandardProperties;

src/types/object.rs

@@ -1,4 +1,5 @@
-use super::Source;
+use super::GetStandardProperties;
+use super::StandardProperties;
 
 #[derive(Debug, PartialEq)]
 pub enum Object<'s> {
@@ -185,197 +186,197 @@ pub struct Timestamp<'s> {
     pub source: &'s str,
 }
 
-impl<'s> Source<'s> for Object<'s> {
-    fn get_source(&'s self) -> &'s str {
+impl<'s> GetStandardProperties<'s> for Object<'s> {
+    fn get_standard_properties(&'s self) -> &'s dyn StandardProperties {
         match self {
-            Object::Bold(obj) => obj.source,
-            Object::Italic(obj) => obj.source,
-            Object::Underline(obj) => obj.source,
-            Object::StrikeThrough(obj) => obj.source,
-            Object::Code(obj) => obj.source,
-            Object::Verbatim(obj) => obj.source,
-            Object::PlainText(obj) => obj.source,
-            Object::RegularLink(obj) => obj.source,
-            Object::RadioLink(obj) => obj.source,
-            Object::RadioTarget(obj) => obj.source,
-            Object::PlainLink(obj) => obj.source,
-            Object::AngleLink(obj) => obj.source,
-            Object::OrgMacro(obj) => obj.source,
-            Object::Entity(obj) => obj.source,
-            Object::LatexFragment(obj) => obj.source,
-            Object::ExportSnippet(obj) => obj.source,
-            Object::FootnoteReference(obj) => obj.source,
-            Object::Citation(obj) => obj.source,
-            Object::CitationReference(obj) => obj.source,
-            Object::InlineBabelCall(obj) => obj.source,
-            Object::InlineSourceBlock(obj) => obj.source,
-            Object::LineBreak(obj) => obj.source,
-            Object::Target(obj) => obj.source,
-            Object::Timestamp(obj) => obj.source,
-            Object::StatisticsCookie(obj) => obj.source,
-            Object::Subscript(obj) => obj.source,
-            Object::Superscript(obj) => obj.source,
+            Object::Bold(inner) => inner,
+            Object::Italic(inner) => inner,
+            Object::Underline(inner) => inner,
+            Object::StrikeThrough(inner) => inner,
+            Object::Code(inner) => inner,
+            Object::Verbatim(inner) => inner,
+            Object::PlainText(inner) => inner,
+            Object::RegularLink(inner) => inner,
+            Object::RadioLink(inner) => inner,
+            Object::RadioTarget(inner) => inner,
+            Object::PlainLink(inner) => inner,
+            Object::AngleLink(inner) => inner,
+            Object::OrgMacro(inner) => inner,
+            Object::Entity(inner) => inner,
+            Object::LatexFragment(inner) => inner,
+            Object::ExportSnippet(inner) => inner,
+            Object::FootnoteReference(inner) => inner,
+            Object::Citation(inner) => inner,
+            Object::CitationReference(inner) => inner,
+            Object::InlineBabelCall(inner) => inner,
+            Object::InlineSourceBlock(inner) => inner,
+            Object::LineBreak(inner) => inner,
+            Object::Target(inner) => inner,
+            Object::StatisticsCookie(inner) => inner,
+            Object::Subscript(inner) => inner,
+            Object::Superscript(inner) => inner,
+            Object::Timestamp(inner) => inner,
         }
     }
 }
 
-impl<'s> Source<'s> for Bold<'s> {
+impl<'s> StandardProperties<'s> for Bold<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Italic<'s> {
+impl<'s> StandardProperties<'s> for Italic<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Underline<'s> {
+impl<'s> StandardProperties<'s> for Underline<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for StrikeThrough<'s> {
+impl<'s> StandardProperties<'s> for StrikeThrough<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Code<'s> {
+impl<'s> StandardProperties<'s> for Code<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Verbatim<'s> {
+impl<'s> StandardProperties<'s> for Verbatim<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for RegularLink<'s> {
+impl<'s> StandardProperties<'s> for RegularLink<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for RadioLink<'s> {
+impl<'s> StandardProperties<'s> for RadioLink<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for RadioTarget<'s> {
+impl<'s> StandardProperties<'s> for RadioTarget<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for PlainLink<'s> {
+impl<'s> StandardProperties<'s> for PlainLink<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for AngleLink<'s> {
+impl<'s> StandardProperties<'s> for AngleLink<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for OrgMacro<'s> {
+impl<'s> StandardProperties<'s> for OrgMacro<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Entity<'s> {
+impl<'s> StandardProperties<'s> for Entity<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for LatexFragment<'s> {
+impl<'s> StandardProperties<'s> for LatexFragment<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for ExportSnippet<'s> {
+impl<'s> StandardProperties<'s> for ExportSnippet<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for FootnoteReference<'s> {
+impl<'s> StandardProperties<'s> for FootnoteReference<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Citation<'s> {
+impl<'s> StandardProperties<'s> for Citation<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for CitationReference<'s> {
+impl<'s> StandardProperties<'s> for CitationReference<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for InlineBabelCall<'s> {
+impl<'s> StandardProperties<'s> for InlineBabelCall<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for InlineSourceBlock<'s> {
+impl<'s> StandardProperties<'s> for InlineSourceBlock<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for LineBreak<'s> {
+impl<'s> StandardProperties<'s> for LineBreak<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Target<'s> {
+impl<'s> StandardProperties<'s> for Target<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for StatisticsCookie<'s> {
+impl<'s> StandardProperties<'s> for StatisticsCookie<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Subscript<'s> {
+impl<'s> StandardProperties<'s> for Subscript<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Superscript<'s> {
+impl<'s> StandardProperties<'s> for Superscript<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for Timestamp<'s> {
+impl<'s> StandardProperties<'s> for Timestamp<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }
 }
 
-impl<'s> Source<'s> for PlainText<'s> {
+impl<'s> StandardProperties<'s> for PlainText<'s> {
     fn get_source(&'s self) -> &'s str {
         self.source
     }

src/types/source.rs

@@ -1,6 +1,3 @@
-pub trait Source<'s> {
-    fn get_source(&'s self) -> &'s str;
-}
 pub(crate) trait SetSource<'s> {
     fn set_source(&mut self, source: &'s str);
 }

src/types/standard_properties.rs

@@ -0,0 +1,58 @@
+// TODO: What is an anonymous AST node and how can I trigger one?
+pub trait StandardProperties<'s> {
+    /// Get the slice of the entire AST node.
+    ///
+    /// This corresponds to :begin to :end in upstream org-mode's standard properties.
+    fn get_source(&'s self) -> &'s str;
+
+    // Get the slice of the AST node's contents.
+    //
+    // This corresponds to :contents-begin to :contents-end
+    // fn get_contents(&'s self) -> &'s str;
+}
+
+// TODO: Write some debugging code to alert when any of the unknown fields below are non-nil in our test data so we can see what these fields represent.
+
+// Order of upstream org-mode's standard properties array:
+//
+// :begin :post-affiliated :contents-begin :contents-end :end :post-blank :secondary :mode :granularity :cached :org-element--cache-sync-key :robust-begin :robust-end :true-level :buffer :deferred :structure :parent
+//
+// Per-field notes: (Leading character: 'X' for not going to include, 'Y' for going to include but not included yet ("Yes"), 'D' for already included ("Done"), '?' for undecided)
+//
+// D :begin - Number of characters (NOT bytes!) since the beginning of the file.
+//
+// ? :post-affiliated - ?
+//
+// Y :contents-begin - Number of characters (NOT bytes!) since the beginning of the file.
+//
+// Y :contents-end - Number of characters (NOT bytes!) since the beginning of the file.
+//
+// D :end - Number of characters (NOT bytes!) since the beginning of the file.
+//
+// Y :post-blank - Number of characters after :contents-end but before :end. This is the trailing whitespace.
+//
+// X :secondary - List of properties that may contain AST nodes. This will be important to reference for implementing TokenIter properly, but I see no value in including this in the StandardProperties trait since which properties contain AST nodes will be self-evident in the struct definition.
+//
+// ? :mode - ?
+//
+// ? :granularity - ?
+//
+// X :cached - ? Based on the name, I'm guessing this is a runtime-optimization rather than something relevant to export from a parser, so (unless I'm wrong about the purpose) I see no reason to include this.
+//
+// X :org-element--cache-sync-key - ? Based on the name, I'm guessing this is a runtime-optimization rather than something relevant to export from a parser, so (unless I'm wrong about the purpose) I see no reason to include this.
+//
+// ? :robust-begin - ? uhh what? What makes this begin/end "robust" and the others not? I have no idea.
+//
+// ? :robust-end - ? uhh what? What makes this begin/end "robust" and the others not? I have no idea.
+//
+// ? :true-level - This seems to correspond to the REAL star count for headlines (as opposed to the headline level we set for when "odd" is enabled instead of the default "oddeven"). This is great information to have, but is this a "standard" property? Does anything other than headlines have this set? I don't know, so I need to investigate. If it is headline-specific then we will not be including this in the StandardProperties trait even though it is in the :standard-properties array in org-mode.
+//
+// X :buffer - This is the Emacs buffer name containing the org-mode document. This seems more like a runtime thing than something we would want to export from our parser so this will not be included.
+//
+// X :deferred - Seems to be a runtime optimization about only calculating some properties when requested.
+//
+// ? :structure - ?
+//
+// X :parent - Some weird numeric reference to the containing object. Since we output a tree structure, I do not see any value in including this, especially considering the back-references would be a nightmare in rust.
+
+// Special case: Plain text. Plain text counts :begin and :end from the start of the text (so :begin is always 0 AFAICT) and instead of including the full set of standard properties, it only includes :begin, :end, and :parent.