org-mode/org-babel-requirements.org

Org-Babel Requirements

This file contains the initial discussion of the requirements for org-babel.

Overview

This project is basically about putting source code into org files. This isn't just code to look pretty as a source code example, but code to be evaluated. Org files have 3 main export targets: org, html and latex. Once we have implemented a smooth bi-directional flow of data between org-mode formats (including tables, and maybe lists and property values) and source-code blocks, we will be able to use org-mode's built in export to publish the results of evaluated source code in any org-supported format using org-mode as an intermediate format. We have a current focus on R code, but we are regarding that more as a working example than as a defining feature of the project.

The main objectives of this project are…

• evaluation of embedded source code

  • execution on demand and on export
  • source blocks
  • header arguments
  • inline source evaluation
  • included source file evaluation ?? maybe
  • caching of evaluation
• interaction with the source-code's process

• output of code evaluation

  • textual/numeric output
  • graphical output
  • non-graphics file creation
  • side effects

• reference to data and evaluation results

  • reference format

  • source-target pairs

    • source block output from org tables
    • source block outpt from other source block
    • source block output from org list ?? maybe
    • org table from source block
    • org table from org table
    • org properties from source block
    • org properties from org table
• export

Objectives and Specs

evaluation of embedded source code

execution on demand and on export

Let's use an asterisk to indicate content which includes the result of code evaluation, rather than the code itself. Clearly we have a requirement for the following transformation:

org → org*

Let's say this transformation is effected by a function `org-eval-buffer'. This transformation is necessary when the target format is org (say you want to update the values in an org table, or generate a plot and create an org link to it), and it can also be used as the first step by which to reach html and latex:

org → org* → html

org → org* → latex

Thus in principle we can reach our 3 target formats with `org-eval-buffer', `org-export-as-latex' and `org-export-as-html'.

An extra transformation that we might want is

org → latex

I.e. export to latex without evaluation of code, in such a way that R code can subsequently be evaluated using Sweave(driver=RweaveLatex), which is what the R community is used to. This would provide a `bail out' avenue where users can escape org mode and enter a workflow in which the latex/noweb file is treated as source.

How do we implement `org-eval-buffer'?

AIUI The following can all be viewed as implementations of org-eval-buffer for R code:

(see this question again posed in org-babel-R.el)

org-eval-light

This is the beginnings of a general evaluation mechanism, that could evaluate python, ruby, shell, perl, in addition to R. The header says it's based on org-eval

what is org-eval??

org-eval was written by Carsten. It lives in the org/contrib/lisp directory because it is too dangerous to include in the base. Unlike org-eval-light org-eval evaluates all source blocks in an org-file when the file is first opened, which could be a security nightmare for example if someone emailed you a pernicious file.

org-R

This accomplishes org → org* in elisp by visiting code blocks and evaluating code using ESS.

RweaveOrg

This accomplishes org → org* using R via

Sweave("file-with-unevaluated-code.org", driver=RweaveOrg, syntax=SweaveSyntaxOrg)

org-exp-blocks.el

Like org-R, this achieves org → org* in elisp by visiting code blocks and using ESS to evaluate R code.

source blocks

(see Special editing and evaluation of source code)

header arguments

(see block headers/parameters)

There are going to be many cases where we want to use header arguments to change the evaluation options of source code, to pass external information to a block of source code and control the inclusion of evaluation results.

inline source evaluation

included source file evaluation

It may be nice to be able to include an entire external file of source code, and then evaluate and export that code as if it were in the file. The format for such a file inclusion could optionally look like the following

#+include_src filename header_arguments

caching of evaluation

Any kind of code that can have a block evaluated could optionally define a function to write the output to a file, or to serialize the output of the function. If a document or block is configured to cache input, write all cached blocks to their own files and either a) hash them, or

1. let git and org-attach track them. Before a block gets eval'd, we

check to see if it has changed. If a document or block is configured to cache output and a print/serialize function is available, write the output of each cached block to its own file. When the file is eval'd and some sort of display is called for, only update the display if the output has changed. Each of these would have an override, presumably something like (… & force) that could be triggered with a prefix arg to the eval or export function.

For R, I would say

;; fake code that only pretends to work
(add-hook 'rorg-store-output-hook 
    '("r" lambda (block-environment block-label)
        (ess-exec (concat "save.image("
                          block-environment
                          ", file = " block-label
                          ".Rdata, compress=TRUE)"))))

The idea being that for r blocks that get eval'd, if output needs to be stored, you should write the entire environment that was created in that block to an Rdata file.

(see block scoping)

interaction with the source-code's process

We should settle on a uniform API for sending code and receiving output from a source process. Then to add a new language all we need to do is implement this API.

for related notes see (Interaction with the R process)

output of code evaluation

textual/numeric output

We (optionally) incorporate the text output as text in the target document

graphical output

We either link to the graphics or (html/latex) include them inline.

I would say, if the block is being evaluated interactively then lets pop up the image in a new window, and if it is being exported then we can just include a link to the file which will be exported appropriately by org-mode.

non-graphics files

? We link to other file output

side effects

If we are using a continuous process in (for example an R process handled by ESS) then any side effects of the process (for example setting values of R variables) will be handled automatically

Are there side-effects which need to be considered aside from those internal to the source-code evaluation process?

reference to data and evaluation results

I think this will be very important. I would suggest that since we are using lisp we use lists as our medium of exchange. Then all we need are functions going converting all of our target formats to and from lists. These functions are already provided by for org tables.

It would be a boon both to org users and R users to allow org tables to be manipulated with the R programming language. Org tables give R users an easy way to enter and display data; R gives org users a powerful way to perform vector operations, statistical tests, and visualization on their tables.

This means that we will need to consider unique id's for source blocks, as well as for org tables, and for any other data source or target.

Implementations

naive

Naive implementation would be to use (org-export-table "tmp.csv") and (ess-execute "read.csv('tmp.csv')").

org-R

org-R passes data to R from two sources: org tables, or csv files. Org tables are first exported to a temporary csv file using org-R-export-to-csv.

org-exp-blocks

org-exp-blocks uses /mirror/org-mode/src/commit/e842d3c36b63e3fb378a67943f207c3380902a53/org-interblock-R-command-to-string to send commands to an R process running in a comint buffer through ESS. org-exp-blocks has no support for dumping table data to R process, or vice versa.

RweaveOrg

NA

reference format

This will be tricky, Dan has already come up with a solution for R, I need to look more closely at that and we should try to come up with a formats for referencing data from source-code in such a way that it will be as source-code-language independent as possible.

Org tables already have a sophisticated reference system in place that allows referencing table ranges in other files, as well as specifying constants in the header arguments of a table. This is described in info:org:References.

Dan: thinking aloud re: referencing data from R

Suppose in some R code, we want to reference data in an org table. I think that requires the use of 'header arguments', since otherwise, under pure evaluation of a code block without header args, R has no way to locate the data in the org buffer. So that suggests a mechanism like that used by org-R whereby table names or unique entry IDs are used to reference org tables (and indeed potentially row/column ranges within org tables, although that subsetting could also be done in R).

Specifically what org-R does is write the table to a temp csv file, and tell R the name of that file. However:

1. We are not limited to a single source of input; the same sort of thing could be done for several sources of input
2. I don't think we even have to use temp files. An alternative would be to have org pass the table contents as a csv-format string to textConnection() in R, thus creating an arbitrary number of input objects in the appropriate R environment (scope) from which the R code can read data when necessary.

That suggests a header option syntax something like

'(:R-obj-name-1 tbl-name-or-id-1 :R-obj-name-2 tbl-name-or-id-2)

As a result of passing that option, the code would be able to access the data referenced by table-name-or-id-2 via read.table(R-obj-name-1).

An extension of that idea would be to allow remote files to be used as data sources. In this case one might need just the remote file (if it's a csv file), or if it's an org file then the name of the file plus a table reference within that org file. Thus maybe something like

'((R-obj-name-1 . (:tblref tbl-name-or-id-1 :file file-1))
  (R-obj-name-2 . (:tblref tbl-name-or-id-2 :file file-2)))

Eric: referencing data in general

So here's some thoughts for referencing data (henceforth referred to as resources). I think this is the next thing we need to tackle for implementation to move forward. We don't need to implement everything below right off the bat, but I'd like to get these lists as full as possible so we don't make any implementation assumptions which preclude real needs.

We need to reference resources of the following types…

• table (list)
• output from a source code block (list or hash)
• property values of an outline header (hash)
• list (list)
• description list (hash)
• more?…

All of these resources will live in org files which could be

• the current file (default)
• another file on the same system (path)
• another file on the web (url)
• another file in a git repo (file and commit hash)

What information should each of these resources be able to supply? I'm thinking (again not that we'll implement all of these but just to think of them)…

• ranges or points of vector data
• key/value pairs from a hash
• when the object was last modified
• commit info (author, date, message, sha, etc…)
• pointers to the resources upon which the resource relies

So we need a referencing syntax powerful enough to handle all of these alternatives. Maybe something like path:sha:name:range where

path

is empty for the current file, is a path for files on the same system, and is a url otherwise

sha

is an option git commit indicator

name

is the table/header/source-block name or id for location inside of the org file (this would not be optional)

range

would indicate which information is requested from the resource, so it could be a range to access parts of a table, or the names of properties to be referenced from an outline header

Once we agree on how this should work, I'll try to stub out some code, so that we can get some simple subset of this functionality working, hopefully something complex enough to do the following…

• resource-reference-example

questions

multiple outputs

Do we want things like a source code block to leave multiple outputs, or do we only want them to be able to output one vector or hash?

environment (state and side-effects)

This design assumes that any changes will explicitly pass data in a functional programming style. This makes no assumptions about things like source code blocks changing state (in general state changes lead to more difficult debugging).

• Do we want to take steps so ensure we do things like execute consecutive R blocks in different environment, or do we want to allow state changes?
• Does this matter?

passing arguments to resources

So I(eric) may be getting ahead of myself here, but what do you think about the ability to pass arguments to resources. I'm having visions of google map-reduce, processes spread out across multiple machines.

Maybe we could do this by allowing the arguments to be specified?

source-target pairs

The following can be used for special considerations based on source-target pairs

Dan: I don't quite understand this subtree; Eric – could you give a little more explanation of this and of your comment above regarding using /mirror/org-mode/src/commit/e842d3c36b63e3fb378a67943f207c3380902a53/lists%20as%20our%20medium%20of%20exchange?

source block output from org tables

source block outpt from other source block

source block output from org list

org table from source block

org table from org table

org properties from source block

org properties from org table

export

once the previous objectives are met export should be fairly simple. Basically it will consist of triggering the evaluation of source code blocks with the org-export-preprocess-hook.

This block export evaluation will be aware of the target format through the htmlp and latexp variables, and can then create quoted #+begin_html and #+begin_latex blocks appropriately.

There will also need to be a set of header arguments related to code export. These would be similar to the results header arguments but would apply to how to handle execution and results during export.

Notes

Block Formats

Unfortunately org-mode how two different block types, both useful. In developing RweaveOrg, a third was introduced.

Eric is leaning towards using the #+begin_src blocks, as that is really what these blocks contain: source code. Austin believes that specifying export options at the beginning of a block is useful functionality, to be preserved if possible.

Note that upper and lower case are not relevant in block headings.

PROPOSED block format

I (Eric) propose that we use the syntax of source code blocks as they currently exist in org-mode with the addition of evaluation, header-arguments, exportation, single-line-blocks, and references-to-table-data.

1. evaluation: These blocks can be evaluated through \C-c\C-c with a slight addition to the code already present and working in org-eval-light.el. All we should need to add for R support would be an appropriate entry in /mirror/org-mode/src/commit/e842d3c36b63e3fb378a67943f207c3380902a53/org-eval-light-interpreters with a corresponding evaluation function. For an example usinga org-eval-light see /mirror/org-mode/src/commit/e842d3c36b63e3fb378a67943f207c3380902a53/%2A%20src%20block%20evaluation%20w/org-eval-light.
2. header-arguments: These can be implemented along the lines of Austin's header arguments in org-sweave.el.
3. exportation: Should be as similar as possible to that done by Sweave, and hopefully can re-use some of the code currently present in org-exp-blocks.el.
4. single-line-blocks: It seems that it is useful to be able to place a single line of R code on a line by itself. Should we add syntax for this similar to Dan's #+RR: lines? I would lean towards something here that can be re-used for any type of source code in the same manner as the #+begin_src R blocks, maybe #+src_R? Dan: I'm fine with this, but don't think single-line blocks are a priority. My #+R lines were something totally different: an attempt to have users specify R code implicitly, using org-mode option syntax.
5. references-to-table-data: I get this impression that this is vital to the efficient use of R code in an org file, so we should come up with a way to reference table data from a single-line-block or from an R source-code block. It looks like Dan has already done this in org-R.el.

Syntax

Multi-line Block

#+begin_src lang header-arguments
 body
#+end

lang

the language of the block (R, shell, elisp, etc…)

header-arguments

a list of optional arguments which control how the block is evaluated and exported, and how the results are handled

body

the actual body of the block

Single-line Block

#+begin_src lang body

• It's not clear how/if we would include header-arguments into a single line block. Suggestions? Can we just leave them out? Dan: I'm not too worried about single line blocks to start off with. Their main advantage seems to be that they save 2 lines. Eric: Fair enough, lets not worry about this now, also I would guess that any code simple enough to fit on one line wouldn't need header arguments anyways.

Include Block

#+include_src lang filename header-arguments

• I think this would be useful, and should be much more work (Dan: didn't get the meaning of that last clause!?). Eric: scratch that, I meant "shouldn't be too much work" :) That way whole external files of source code could be evaluated as if they were an inline block. Dan: again I'd say not a massive priority, as I think all the languages we have in mind have facilities for doing this natively, thus I think the desired effect can often be achieved from within a

  to include we shouldn't wast too much effort on it in the beginning.
  
  What do you think?  Does this accomplish everything we want to be able
  to do with embedded R source code blocks?
  
  ***** src block evaluation w/org-eval-light
  here's an example using org-eval-light.el
  
  first load the org-eval-light.el file
  
  [[elisp:(load (expand-file-name "org-eval-light.el" (expand-file-name "existing_tools" (file-name-directory buffer-file-name))))]]
  
  then press =\C-c\C-c= inside of the following src code snippet.  The
  results should appear in a comment immediately following the source
  code block.  It shouldn't be too hard to add R support to this
  function through the `org-eval-light-interpreters' variable.
  
  (Dan: The following causes error on export to HTML hence spaces inserted at bol)
  
  #+begin_src shell
  date

existing formats

Source code blocks

Org has an extremely useful method of editing source code and examples in their native modes. In the case of R code, we want to be able to use the full functionality of ESS mode, including interactive evaluation of code.

Source code blocks look like the following and allow for the special editing of code inside of the block through `org-edit-special'.

,## hit C-c ' within this block to enter a temporary buffer in r-mode.

,## while in the temporary buffer, hit C-c C-c on this comment to
,## evaluate this block
a <- 3
a

,## hit C-c ' to exit the temporary buffer

dblocks

dblocks are useful because org-mode will automatically call `org-dblock-write:dblock-type' where dblock-type is the string following the #+BEGIN: portion of the line.

dblocks look like the following and allow for evaluation of the code inside of the block by calling \C-c\C-c on the header of the block.

R blocks

In developing RweaveOrg, Austin created org-sweave.el. This allows for the kind of blocks shown in testing.Rorg. These blocks have the advantage of accepting options to the Sweave preprocessor following the #+BEGIN_R declaration.

block headers/parameters

Regardless of the syntax/format chosen for the source blocks, we will need to be able to pass a list of parameters to these blocks. These should include (but should certainly not be limited to)

label or id

Label of the block, should we provide facilities for automatically generating a unique one of these?

file

names of file to which graphical/textual/numerical/tabular output should be written. Do we need this, or should this be controlled through the source code itself?

results

indication of where the results should be placed, maybe the following values…

append

default meaning just append to the current buffer immediately following the current source block

replace

like append, but replace any results currently there

file

save the results in a new file, and place a link to the file into the current buffer immediately following the source code block

table

save the results into a table, maybe use a table id:range to identify which table and where therein

nil

meaning just discard the results

not sure of a good name here

flags for when/if the block should be evaluated (on export etc…)

again can't thing of a concise name

flags for how the results of the export should be displayed/included

scope

flag indicating whether the block should have a local or global scope

?

flags specific to the language of the source block

?

etc…

I think fleshing out this list is an important next step.

Interaction with the R process

We should take care to implement this in such a way that all of the different components which have to interactive with R including:

• evaluation of source code blocks
• automatic evaluation on export
• evaluation of \R{} snippets
• evaluation of single source code lines
• evaluation of included source code files
• sending/receiving vector data

I think we currently have two implementations of interaction with R processes; org-R.el and org-exp-blocks.el. We should be sure to take the best of each of these approaches.

More on the exchange of data at between org-mode and source code blocks at reference to data and evaluation results.

block scoping

(see caching of evaluation)

This inadvertently raises the issue of scoping. The pretend function pretends that we will create a block-local scope, and that we can save just the things in that scope. Sweave takes the make-everything-global approach. I can see advantages either way. If we make block-local scopes, we can save each one independently, and generally speaking it seems like more granularity==more control. If we make everything global, we can refer to entities declared in earlier blocks without having to explicitly import those entities into the current block. I think this counts in the "need to think about it early on" category.

If we did want block-local scopes, in R we can start every eval with something like

;; fake code that pretends to create a new, empty environment (ess-exec (concat block-env " <- new.env()")) (ess-exec (concat "eval(" block-contents ", envir=" block-env ")"))

If we decide we want block-scoping, I'm sure Dan and I can figure out the right way to do this in R, if he hasn't already. I haven't thought at all about how these scope issues generalize to, say, bash blocks.

Maybe this is something that should be controlled by a header argument?

\C-c\C-c evaluation

With org-mode version at least 6.23, see the documentation for info:org:Context-sensitive commands.

free explicit variables

Maybe we should have some idea of variables independent of any particular type of source code or source block. These could be variables that have a value inside of the scope of the org-mode file, and they could be used as a transport mechanism for information transfer between org-tables, org-lists, and different source-blocks.

Each type of source code (and org-mode types like tables, lists, etc…) would need to implement functions for converting different types of data to and from these variables (which would be elisp variables).

So for example say we want to read the values from a table into an R block, perform some calculations, and then write the results back into the table. We could

1. assign the table to a variable

   • the table would be converted into a lisp vector (list of lists)
   • the vector would be saved in the variable

2. an R source block would reference the variable

   • the variable would be instantiated into an R variable (through mechanisms mentioned elsewhere)
   • the R code is executed, and the value of the variable inside of R is updated
   • when the R block finished the value of the variable globally in the org buffer would be updated
3. optionally the global value of the variable would be converted back into an org-mode table and would be used to overwrite the existing table.

What do you think?

This might not be too different from what we were already talking about, but I think the introduction of the idea of having variables existing independently of any tables or source code blocks is novel and probably has some advantages (and probably shortfalls).

COMMENT Commentary

I'm seeing this as like commit notes, and a place for less formal communication of the goals of our changes.

Eric <2009-02-06 Fri 15:41>

I think we're getting close to a comprehensive set of objectives (although since you two are the real R user's I leave that decision up to you). Once we've agreed on a set of objectives and agreed on at least to broad strokes of implementation, I think we should start listing out and assigning tasks.

Eric <2009-02-09 Mon 14:25>

I've done a fairly destructive edit of this file. The main goal was to enforce a structure on the document that we can use moving forward, so that any future objective changes are all made to the main objective list.

I apologize for removing sections written by other people. I did this when they were redundant or it was not clear how to fit them into this structure. Rest assured if the previous text wasn't persisted in git I would have been much more cautious about removing it.

I hope that this outline structure should be able to remain stable through the process of fleshing out objectives, and cashing those objectives out into tasks. That said, please feel free to make any changes that you see fit.

Dan <2009-02-12 Thu 10:23>

Good job Eric with major works on this file.

Eric <2009-02-22 Sun 13:17>

So I skipped ahead and got started on the fun part. Namely stubbing out some of the basic functionality. Please don't take any of the decisions I've made so far (on things like names, functionality, design etc…) as final decisions, I'm of course open to and hoping for improvement.

So far org-babel.el and org-babel-script.el can be used to evaluate source code blocks of simple scripting languages. It shouldn't be too hard (any takers) to write a org-babel-R.el modeled after org-babel-script.el to use for evaluating R code files.

See the Sandbox for evaluable examples.

Eric <2009-02-23 Mon 15:12>

While thinking about how to implement the transfer of data between source blocks and the containing org-mode file, I decided it might be useful to explicitly support the existence of variables which exist independent of source blocks or tables. I'd appreciate any feedback… (see free explicit variables)

Eric <2009-02-23 Mon 17:53>

So as I start populating this file with source code blocks I figure I should share this… I don't know if you guys use yasnippet at all, but if you do you might find this block-snippet org-mode snippet useful (I use it all the time).

Dan <2009-05-14 Thu 19:13>

Please note that I am at an early stage of learning org-babel / studying the code so I may be misunderstanding things.

1. removed prefix arg here
2. changed :replace -> replace
3. added some tasks and bugs

Buffer Dictionary

LocalWords: DBlocks dblocks org-babel el eric fontification