From aa1747b0656edf50691dd7ebae7c6ab9789f25f9 Mon Sep 17 00:00:00 2001 From: Dan Davison Date: Sun, 19 Jul 2009 13:56:38 -0400 Subject: [PATCH] Quite a few revisions / additions to the tasks and bugs lists. --- lisp/langs/org-babel-gnuplot.el | 6 +- org-babel.org | 475 ++++++++++++++++++++++---------- 2 files changed, 331 insertions(+), 150 deletions(-) diff --git a/lisp/langs/org-babel-gnuplot.el b/lisp/langs/org-babel-gnuplot.el index 3f587ed8b..766c27a7c 100644 --- a/lisp/langs/org-babel-gnuplot.el +++ b/lisp/langs/org-babel-gnuplot.el @@ -48,9 +48,11 @@ (defun org-babel-execute:gnuplot (body params) "Execute a block of Gnuplot code with org-babel. This function is -called by `org-babel-execute-src-block'." +called by `org-babel-execute-src-block' via multiple-value-bind." (message "executing Gnuplot source code block") - (let* ((vars (org-babel-ref-variables params)) + (let* (;; should be able to lose the next two lines but I don't know + ;; how to test it so not doing it now. + (vars (org-babel-ref-variables params)) (result-params (split-string (or (cdr (assoc :results params)) ""))) (out-file (cdr (assoc :file params))) (cmdline (cdr (assoc :cmdline params))) diff --git a/org-babel.org b/org-babel.org index 7af3f8c00..9d78445d9 100644 --- a/org-babel.org +++ b/org-babel.org @@ -207,7 +207,24 @@ would then be [[#sandbox][the sandbox]]. #+end_src -* Tasks [32/50] +* Tasks [35/55] +** PROPOSED allow `anonymous' function block with function call args? + My question here is simply whether we're going to allow +#+begin_src python(arg=ref) +# whatever +#+end_src + +but with preference given to +#+srcname blockname(arg=ref) +** PROPOSED allow :result as synonym for :results? +** PROPOSED allow 'output mode to return stdout as value? + Maybe we should allow this. In fact, if block x is called + with :results output, and it references blocks y and z, then + shouldn't the output of x contain a concatenation of the outputs of + y and z, together with x's own output? That would raise the + question of what happens if y is defined with :results output and z + with :results value. I guess z's (possibly vector/tabular) output + would be inside a literal example block containing the whole lot. ** PROPOSED optional timestamp for output Add option to place an (inactive) timestamp at the #+resname, to record when that output was generated. @@ -357,6 +374,13 @@ org-mode core similar status to a source code block? - Would be nice to allow org and non-org files to be remote ** TODO figure out how to handle errors during evaluation + I expect it will be hard to do this properly, but ultimately it + would be nice to be able to specify somewhere to receive STDERR, + and to be warned if it is non-empty. + + Probably simpler in non-session evaluation than session? At least + the mechanism will be different I guess. + R has a try function, with error handling, along the lines of python. I bet ruby does too. Maybe more of an issue for functional style; in my proposed scripting style the error just gets dumped to @@ -532,17 +556,86 @@ we should color these blocks differently *** TODO refine html exportation should use a span class, and should show original source in tool-tip -** STARTED Column (and row) names of tables in R input/output -*** DONE column names - This has been implemented: Automatic on input to R; optional in output. -*** TODO row names - Perhaps add a :rownames header arg. This would be an integer +** TODO LoB: re-implement plotting and analysis functions from org-R + I'll do this soon, now that we things are a bit more settled and we + have column names in R. +** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format + I haven't thought about this properly. Just noting it down. What + Sweave uses is called "R noweb" (.Rnw). + + I found a good description of noweb in the following article (see + the [[http://www.cs.tufts.edu/~nr/pubs/lpsimp.pdf][pdf]]). + + I think there are two parts to noweb, the construction of + documentation and the extraction of source-code (with notangle). + + *documentation*: org-mode handles all of our documentation needs in + a manner that I believe is superior to noweb. + + *source extraction* At this point I don't see anyone writing large + applications with 100% of the source code contained in org-babel + files, rather I see org-babel files containing things like + - notes with active code chunks + - interactive tutorials + - requirements documents with code running test suites + - and of course experimental reports with the code to run the + experiment, and perform analysis + + Basically I think the scope of the programs written in org-babel + (at least initially) will be small enough that it wont require the + addition of a tangle type program to extract all of the source code + into a running application. + + On the other hand, since we already have named blocks of source + code which reference other blocks on which they rely, this + shouldn't be too hard to implement either on our own, or possibly + relying on something like noweb/notangle. + +** PROPOSED support for passing paths to files between source blocks +Maybe this should be it's own result type (in addition to scalars and +vectors). The reason being that some source-code blocks (for example +ditaa or anything that results in the creation of a file) may want to +pass a file path back to org-mode which could then be inserted into +the org-mode buffer as a link to the file... + +This would allow for display of images upon export providing +functionality similar to =org-exp-blocks= only in a more general +manner. +** DEFERRED Support rownames and other org babel table features? + + The full org table features are detailed in the manual [[http://orgmode.org/manual/Advanced-features.html#Advanced-features][here]]. + +*** rownames + Perhaps add a :rownames header arg. This would be an integer (usually 1) which would have the effect of post-processing all the variables created in the R session in the following way: if the integer is j, set the row names to the contents of column j and delete column j. Perhaps it is artificial to allow this integer to take any value other than 1. The default would be nil which would mean no such behaviour. + + Actually I don't know about that. If multiple variables are passed + in, it's not appropriate to alter them all in the same way. The + rownames specification would normally refer to just one of the + variables. For now maybe just say this has to be done in R. E.g. + +#+TBLNAME: sample-sizes + | collection | size | exclude | include | exclude2 | include2 | + |-----------------+------+---------+---------+----------+----------| + | 58C | 2936 | 8 | 2928 | 256 | 2680 | + | MS | 5852 | 771 | 5081 | 771 | 5081 | + | NBS | 2929 | 64 | 2865 | 402 | 2527 | + | POBI | 2717 | 1 | 2716 | 1 | 2716 | + | 58C+MS+NBS+POBI | | | 13590 | | 13004 | +#+TBLFM: @2$4=@2$2 - @2$3::@2$6=@2$2 - @2$5::@3$4=@3$2-@3$3::@3$6=@3$2 - @3$5::@4$4=@4$2 - @4$3::@4$6=@4$2 - @4$5::@5$4=@5$2-@5$3::@5$6=@5$2 - @5$5::@6$4=vsum(@2$4..@5$4)::@6$6=vsum(@2$6..@5$6) + +#+srcname: make-size-table(size=sample-sizes) +#+begin_src R + rownames(size) <- size[,1] + size <- size[,-1] +#+end_src + + *** Old notes [I don't think it's as problematic as this makes out] This is non-trivial, but may be worth doing, in particular to @@ -640,106 +733,6 @@ tabel Another example is in the [[*operations%20in%20on%20tables][grades example]]. -** TODO re-implement helper functions from org-R -*** Initial statement [Eric] - Much of the power of org-R seems to be in it's helper functions for - the quick graphing of tables. Should we try to re-implement these - functions on top of org-babel? - - I'm thinking this may be useful both to add features to org-babel-R and - also to potentially suggest extensions of the framework. For example - one that comes to mind is the ability to treat a source-code block - like a function which accepts arguments and returns results. Actually - this can be it's own TODO (see [[* source blocks as functions][source blocks as functions]]). -*** Objectives [Dan] - - We want to provide convenient off-the-shelf actions - (e.g. plotting data) that make use of our new code evaluation - environment but do not require any actual coding. -*** Initial Design proposal [Dan] - - *Input data* will be specified using the same mechanism as :var - references, thus the input data may come from a table, or - another source block, and it is initially available as an elisp - data structure. - - We introduce a new #+ line, e.g. #+BABELDO. C-c C-c on that - line will apply an *action* to the referenced data. - - *Actions correspond to source blocks*: our library of available - actions will be a library of org-babel source blocks. Thus the - code for executing an action, and the code for dealing with the - output of the action will be the same code as for executing - source blocks in general - - Optionally, the user can have the relevant source block inserted - into the org buffer after the (say) #+BABELDO line. This will - allow the user to fine tune the action by modifying the code - (especially useful for plots). - - So maybe a #+BABELDO line will have header args - - :data (a reference to a table or source code block) - - :action (or should that be :srcname?) which will be something - like :action pie-chart, referring to a source block which will - be executed with the :data referent passed in using a :var arg. - - :showcode or something controlling whether to show the code - -*** Modification to design - I'm implementing this, at least initially, as a new interpreter - named 'babel', which has an empty body. 'babel' blocks take - a :srcname header arg, and look for the source-code block with - that name. They then execute the referenced block, after first - appending their own header args on to the target block's header - args. - - If the target block is in the library of babel (a.o.t. e.g. the - current buffer), then the code in the block will refer to the - input data with a name dictated by convention (e.g. __data__ - (something which is syntactically legal in all languages...). Thus - the babel block will use a :var __data__ = whatever header arg to - reference the data to be plotted. - -*** Current design - This is covered by the [[file:library-of-babel.org][Library of Babel]], which will contain - ready-made source blocks designed to carry out useful common tasks. - -** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format - I haven't thought about this properly. Just noting it down. What - Sweave uses is called "R noweb" (.Rnw). - - I found a good description of noweb in the following article (see - the [[http://www.cs.tufts.edu/~nr/pubs/lpsimp.pdf][pdf]]). - - I think there are two parts to noweb, the construction of - documentation and the extraction of source-code (with notangle). - - *documentation*: org-mode handles all of our documentation needs in - a manner that I believe is superior to noweb. - - *source extraction* At this point I don't see anyone writing large - applications with 100% of the source code contained in org-babel - files, rather I see org-babel files containing things like - - notes with active code chunks - - interactive tutorials - - requirements documents with code running test suites - - and of course experimental reports with the code to run the - experiment, and perform analysis - - Basically I think the scope of the programs written in org-babel - (at least initially) will be small enough that it wont require the - addition of a tangle type program to extract all of the source code - into a running application. - - On the other hand, since we already have named blocks of source - code which reference other blocks on which they rely, this - shouldn't be too hard to implement either on our own, or possibly - relying on something like noweb/notangle. - -** PROPOSED support for passing paths to files between source blocks -Maybe this should be it's own result type (in addition to scalars and -vectors). The reason being that some source-code blocks (for example -ditaa or anything that results in the creation of a file) may want to -pass a file path back to org-mode which could then be inserted into -the org-mode buffer as a link to the file... - -This would allow for display of images upon export providing -functionality similar to =org-exp-blocks= only in a more general -manner. - ** DEFERRED use textConnection to pass tsv to R? When passing args from the org buffer to R, the following route is used: arg in buffer -> elisp -> tsv on file -> data frame in R. I @@ -932,6 +925,70 @@ to the command if BUFF is not given.) 2) The function is called inside of a =write.table= function call writing the results to a table 3) The table is read using =org-table-import= + +** DONE extensible library of callable source blocks +*** Current design + This is covered by the [[file:library-of-babel.org][Library of Babel]], which will contain + ready-made source blocks designed to carry out useful common tasks. +*** Initial statement [Eric] + Much of the power of org-R seems to be in it's helper functions for + the quick graphing of tables. Should we try to re-implement these + functions on top of org-babel? + + I'm thinking this may be useful both to add features to org-babel-R and + also to potentially suggest extensions of the framework. For example + one that comes to mind is the ability to treat a source-code block + like a function which accepts arguments and returns results. Actually + this can be it's own TODO (see [[* source blocks as functions][source blocks as functions]]). +*** Objectives [Dan] + - We want to provide convenient off-the-shelf actions + (e.g. plotting data) that make use of our new code evaluation + environment but do not require any actual coding. +*** Initial Design proposal [Dan] + - *Input data* will be specified using the same mechanism as :var + references, thus the input data may come from a table, or + another source block, and it is initially available as an elisp + data structure. + - We introduce a new #+ line, e.g. #+BABELDO. C-c C-c on that + line will apply an *action* to the referenced data. + - *Actions correspond to source blocks*: our library of available + actions will be a library of org-babel source blocks. Thus the + code for executing an action, and the code for dealing with the + output of the action will be the same code as for executing + source blocks in general + - Optionally, the user can have the relevant source block inserted + into the org buffer after the (say) #+BABELDO line. This will + allow the user to fine tune the action by modifying the code + (especially useful for plots). + - So maybe a #+BABELDO line will have header args + - :data (a reference to a table or source code block) + - :action (or should that be :srcname?) which will be something + like :action pie-chart, referring to a source block which will + be executed with the :data referent passed in using a :var arg. + - :showcode or something controlling whether to show the code + +*** Modification to design + I'm implementing this, at least initially, as a new interpreter + named 'babel', which has an empty body. 'babel' blocks take + a :srcname header arg, and look for the source-code block with + that name. They then execute the referenced block, after first + appending their own header args on to the target block's header + args. + + If the target block is in the library of babel (a.o.t. e.g. the + current buffer), then the code in the block will refer to the + input data with a name dictated by convention (e.g. __data__ + (something which is syntactically legal in all languages...). Thus + the babel block will use a :var __data__ = whatever header arg to + reference the data to be plotted. + +** DONE Column names in R input/output + This has been implemented: Automatic on input to R; optional in + output. Note that this equates column names with the header row in + an org table; whereas org actually has a mechanism whereby a row + with a '!' in the first field defines column names. I have not + attempted to support these org table mechanisms yet. See [[*Support%20rownames%20and%20other%20org%20babel%20table%20features][this + DEFERRED todo item]]. ** DONE use example block for large amounts of stdout output? We're currently `examplizing' with : at the beginning of the line, but should larger amounts of output be in a @@ -2138,8 +2195,90 @@ to specify a file holding the results (see [[* file result types][file result types]]) -* Bugs [22/34] +* Bugs [22/36] +** TODO Default args + This would be good thing to address soon. I'm imagining that + e.g. here, the 'caller' block would return the answer 30. I believe + there's a few issues here: i.e. the naked 'a' without a reference + is not understood; the default arg b=6 is not understood. + +#+srcname: adder(a, b=6) +#+begin_src python +a+b +#+end_src + +#+srcname: caller(var=adder(a=24)) +#+begin_src python +var +#+end_src +** TODO function calls in #+srcname: refs + + My srcname references don't seem to be working for function + calls. This needs fixing. + +#+srcname: called +#+begin_src python +56 +#+end_src + +srcname function call doesn't work for calling a source block +#+srcname: caller(var1=called()) +#+begin_src python +var1 +#+end_src + +They do work for a simple reference +#+srcname: caller(var1=56) +#+begin_src python +var1 +#+end_src + +#+resname: caller +: 56 + +#+srcname: caller2 +#+begin_src python :var var1=called() +var1 +#+end_src + +#+resname: +: 56 +** TODO creeping blank lines + There's still inappropriate addition of blank lines in some circumstances. + + Hmm, it's a bit confusing. It's to do with o-b-remove-result. LoB + removes the entire (#+resname and result) and starts from scratch, + whereas #+begin_src only removes the result. I haven't worked out + what the correct fix is yet. Maybe the right thing to do is to make + sure that those functions (o-b-remove-result et al.) are neutral + with respect to newlines. Sounds easy, but... + + E.g. + +#+begin_src sh +b=5 +#+end_src + + + + Compare the results of +#+lob: python-add(a=5, b=17) + +#+resname: python-add(a=5, b=17) +: 22 +-------------------------------- + +#+begin_src python +23 +#+end_src + +#+resname: +: 23 +--------------------- + ** TODO avoid stripping whitespace from output when :results output + This may be partly solved by using o-b-chomp rather than o-b-trim + in the o-b-LANG-evaluate functions. ** TODO problem with newlines in output when :results value #+begin_src python :results value '\n'.join(map(str, range(4))) @@ -2147,6 +2286,87 @@ to specify a file holding the results #+resname: : 0 + +Whereas I was hoping for + +| 0 | +| 1 | +| 2 | +| 3 | + +This is some sort of non-printing char / quoting issue I think. Note +that + +#+begin_src python :results value +'\\n'.join(map(str, range(4))) +#+end_src + +#+resname: +: 0\n1\n2\n3 + +Also, note that +#+begin_src python :results output +print('\n'.join(map(str, range(4)))) +#+end_src + +#+resname: +: 0 +: 1 +: 2 +: 3 + +*** collapsing consecutive newlines in string output + + This is an example of the same bug + +#+srcname: multi-line-string-output +#+begin_src ruby :results output +"the first line ends here + + + and this is the second one + +even a third" +#+end_src + +This doesn't produce anything at all now. I believe that's because +I've changed things so that :results output really does *not* get the +value of the block, only the STDOUT. So if we add a print statement +this works OK. + +#+srcname: multi-line-string-output +#+begin_src ruby :results output +print "the first line ends here + + + and this is the second one + +even a third" +#+end_src + +#+resname: +: the first line ends here +: +: +: and this is the second one +: +: even a third + +However, the behaviour with :results value is wrong + +#+srcname: multi-line-string-value +#+begin_src ruby +"the first line ends here + + + and this is the second one + +even a third" +#+end_src + +#+resname: +: 0 + ** TODO prompt characters appearing in output with R #+begin_src R :session *R* :results output x <- 6 @@ -2179,20 +2399,6 @@ for example #+end_src #+resname: this-doesn't-match-orgtbl - -** TODO collapsing consecutive newlines in string output - -#+srcname: multi-line-string-output -#+begin_src ruby :results output -"the first line ends here - - - and this is the second one - -even a third" -#+end_src - -#+resname: multi-line-string-output ** PROPOSED external shell execution can't isolate return values I have no idea how to do this as of yet. The result is that when @@ -2221,33 +2427,6 @@ the same for the other languages. [Dan] not org-babel-ruby-evaluate ** TODO use new merge function [[file:lisp/org-babel-ref.el::t%20nil%20org%20combine%20plists%20args%20nil][here]]? And at other occurrences of org-combine-plists? -** TODO creeping blank lines - There's still inappropriate addition of blank lines in some circumstances. E.g. - -#+begin_src sh -b=5 -#+end_src - - - Compare the results of -#+lob: python-add(a=5, b=17) - -#+resname: python-add(a=5, b=17) -: 22 --------------------------------- - -#+begin_src python -23 -#+end_src - -#+resname: -: 23 ---------------------- - - Hmm, it's a bit confusing. I think it's to do with the fact that - LoB removes the entire (#+resname and result) and starts from - scratch, whereas #+begin_src only removes the result. I haven't - worked out what the correct fix is yet. ** TODO LoB is not populated on startup org-babel-library-of-babel is nil for me on startup. I have to evaluate the [[file:lisp/org-babel-lob.el::][org-babel-lob-ingest]] line manually.