10.19.2 Info from Han-Wen email (LilyPond Contributor’s Guide)

#### 10.19.2 Info from Han-Wen email

In 2004, Douglas Linhardt decided to try starting a document that would explain LilyPond architecture and design principles. The material below is extracted from that email, which can be found at http://thread.gmane.org/gmane.comp.gnu.lilypond.devel/2992. The headings reflect questions from Doug or comments from Han-Wen; the body text are Han-Wen’s answers.

#### Figuring out how things work.

I must admit that when I want to know how a program works, I use grep and emacs and dive into the source code. The comments and the code itself are usually more revealing than technical documents.

#### What’s a grob, and how is one used?

Graphical object - they are created from within engravers, either as Spanners (derived class) -slurs, beams- or Items (also a derived class) -notes, clefs, etc.

There are two other derived classes System (derived from Spanner, containing a "line of music") and Paper_column (derived from Item, it contains all items that happen at the same moment). They are separate classes because they play a special role in the linebreaking process.

#### What’s a smob, and how is one used?

A C(++) object that is encapsulated so it can be used as a Scheme object. See Guile info, "19.3 Defining New Types (Smobs)"

#### When is each C++ class constructed and used?

- Music classes

  In the parser.yy see the macro calls MAKE_MUSIC_BY_NAME().

- Contexts

  Constructed during "interpreting" phase.

- Engravers

  Executive branch of Contexts, plugins that create grobs, usually one engraver per grob type. Created together with context.

- Layout Objects

  = grobs

- Grob Interfaces

  These are not C++ classes per se. The idea of a Grob interface hasn’t crystallized well. ATM, an interface is a symbol, with a bunch of grob properties. They are not objects that are created or destroyed.

- Iterators

  Objects that walk through different music classes, and deliver events in a synchronized way, so that notes that play together are processed at the same moment and (as a result) end up on the same horizontal position.

  Created during interpreting phase.

  BTW, the entry point for interpreting is ly:run-translator (ly_run_translator on the C++ side)

#### Can you get to Context properties from a Music object?

You can create music object with a Scheme function that reads context properties (the \applycontext syntax). However, that function is executed during Interpreting, so you can not really get Context properties from Music objects, since music objects are not directly connected to Contexts. That connection is made by the Music_iterators

#### Can you get to Music properties from a Context object?

Yes, if you are given the music object within a Context object. Normally, the music objects enter Contexts in synchronized fashion, and the synchronization is done by Music_iterators.

#### What is the relationship between C++ classes and Scheme objects?

Smobs are C++ objects in Scheme. Scheme objects (lists, functions) are manipulated from C++ as well using the Guile C function interface (prefix: scm_)

#### How do Scheme procedures get called from C++ functions?

scm_call_*, where * is an integer from 0 to 4. Also scm_c_eval_string (), scm_eval ()

#### How do C++ functions get called from Scheme procedures?

Export a C++ function to Scheme with LY_DEFINE.

#### What is the flow of control in the program?

Good question. Things used to be clear-cut, but we have Scheme and SMOBs now, which means that interactions do not follow a very rigid format anymore. See below for an overview, though.

#### Does the parser make Scheme procedure calls or C++ function calls?

Both. And the Scheme calls can call C++ and vice versa. It’s nested, with the SCM datatype as lubrication between the interactions

(I think the word "lubrication" describes the process better than the traditional word "glue")

#### How do the front-end and back-end get started?

Front-end: a file is parsed, the rest follows from that. Specifically,

Parsing leads to a Music + Music_output_def object (see parser.yy, definition of toplevel_expression )

A Music + Music_output_def object leads to a Global_context object (see ly_run_translator ())

During interpreting, Global_context + Music leads to a bunch of Contexts (see Global_translator::run_iterator_on_me ()).

After interpreting, Global_context contains a Score_context (which contains staves, lyrics etc.) as a child. Score_context::get_output () spews a Music_output object (either a Paper_score object for notation or Performance object for MIDI).

The Music_output object is the entry point for the backend (see ly_render_output ()).

The main steps of the backend itself are in

- `paper-score.cc` , Paper_score::process_
- `system.cc` , System::get_lines()
- The step, where things go from grobs to output, is in System::get_line(): each grob delivers a Stencil (a Device independent output description), which is interpreted by our outputting backends (`scm/output-tex.scm` and `scm/output-ps.scm`) to produce TeX and PS.

Interactions between grobs and putting things into .tex and .ps files have gotten a little more complex lately. Jan has implemented page-breaking, so now the backend also involves Paper_book, Paper_lines and other things. This area is still heavily in flux, and perhaps not something you should want to look at.

#### How do the front-end and back-end communicate?

There is no communication from backend to front-end. From front-end to backend is simply the program flow: music + definitions gives contexts, contexts yield output, after processing, output is written to disk.

#### Where is the functionality associated with KEYWORDs?

See `my-lily-lexer.cc` (keywords, there aren’t that many) and `ly/*.ly` (most of the other backslashed `/\words` are identifiers)

#### What Contexts/Properties/Music/etc. are available when they are processed?

What do you mean exactly with this question?

See `ly/engraver-init.ly` for contexts, see `scm/define-*.scm` for other objects.

#### How do you decide if something is a Music, Context, or Grob property?

Why is part-combine-status a Music property when it seems (IMO) to be related to the Staff context?

The Music_iterators and Context communicate through two channels

Music_iterators can set and read context properties, idem for Engravers and Contexts

Music_iterators can send "synthetic" music events (which aren’t in the input) to a context. These are caught by Engravers. This is mostly a one way communication channel.

part-combine-status is part of such a synthetic event, used by Part_combine_iterator to communicate with Part_combine_engraver.

#### Deciding between context and music properties

I’m adding a property to affect how \autoChange works. It seems to me that it should be a context property, but the Scheme autoChange procedure has a Music argument. Does this mean I should use a Music property?

\autoChange is one of these extra strange beasts: it requires look-ahead to decide when to change staves. This is achieved by running the interpreting step twice (see `scm/part-combiner.scm` , at the bottom), and storing the result of the first step (where to switch staves) in a Music property. Since you want to influence that where-to-switch list, your must affect the code in make-autochange-music (`scm/part-combiner.scm`). That code is called directly from the parser and there are no official "parsing properties" yet, so there is no generic way to tune \autoChange. We would have to invent something new for this, or add a separate argument,

        \autoChange #around-central-C ..music..

where around-central-C is some function that is called from make-autochange-music.

#### More on context and music properties

From Neil Puttock, in response to a question about transposition:

Context properties (using \set & \unset) are tied to engravers: they provide information relevant to the generation of graphical objects.

Since transposition occurs at the music interpretation stage, it has no direct connection with engravers: the pitch of a note is fixed before a notehead is created. Consider the following minimal snippet:

    { c' }

This generates (simplified) a NoteEvent, with its pitch and duration as event properties,

    (make-music
      'NoteEvent
      'duration
      (ly:make-duration 2 0 1 1)
      'pitch
      (ly:make-pitch 0 0 0)

which the Note_heads_engraver hears. It passes this information on to the NoteHead grob it creates from the event, so the head’s correct position and duration-log can be determined once it’s ready for printing.

If we transpose the snippet,

    \transpose c d { c' }

the pitch is changed before it reaches the engraver (in fact, it happens just after the parsing stage with the creation of a TransposedMusic music object):

    (make-music
     'NoteEvent
     'duration
     (ly:make-duration 2 0 1 1)
     'pitch
     (ly:make-pitch 0 1 0)

You can see an example of a music property relevant to transposition: untransposable.

    \transpose c d { c'2 \withMusicProperty #'untransposable ##t c' }

-> the second c’ remains untransposed.

Take a look at `lily/music.cc` to see where the transposition takes place.

#### How do I tell about the execution environment?

I get lost figuring out what environment the code I’m looking at is in when it executes. I found both the C++ and Scheme autoChange code. Then I was trying to figure out where the code got called from. I finally figured out that the Scheme procedure was called before the C++ iterator code, but it took me a while to figure that out, and I still didn’t know who did the calling in the first place. I only know a little bit about Flex and Bison, so reading those files helped only a little bit.

Han-Wen: GDB can be of help here. Set a breakpoint in C++, and run. When you hit the breakpoint, do a backtrace. You can inspect Scheme objects along the way by doing

    p ly_display_scm(obj)

this will display OBJ through Guile.

#### LilyPond Contributor’s Guide v2.26.0 (stable-branch).

- 1 Introduction to contributing
  - 1.1 Help us
  - 1.2 Overview of work flow
  - 1.3 Summary for experienced developers
  - 1.4 Mentors
- 2 Quick start
  - 2.1 LilyDev
    - Installing LilyDev in VirtualBox
    - Configuring LilyDev in VirtualBox
  - 2.2 Compiling with LilyDev
  - 2.3 Now start work!
- 3 Working with source code
  - 3.1 Setting up
    - 3.1.1 Installing Git
    - 3.1.2 Creating a GitLab account and setting up SSH
    - 3.1.3 Cloning and forking the repository
    - 3.1.4 Configuring Git
  - 3.2 Git cheat sheet
  - 3.3 Lifecycle of a merge request
    - 3.3.1 Uploading a patch for review
    - 3.3.2 Automated testing
    - 3.3.3 Patch countdown
    - 3.3.4 Merging to master
    - 3.3.5 Abandoned patches
  - 3.4 Writing good commit messages
  - 3.5 Commit access
  - 3.6 Further Git documentation resources
  - 3.7 Repository directory structure
- 4 Compiling
  - 4.1 Overview of compiling
  - 4.2 Requirements
    - 4.2.1 Requirements for running LilyPond
    - 4.2.2 Requirements for compiling LilyPond
      - Fedora
      - Linux Mint
      - OpenSUSE
      - Ubuntu
      - Other
    - 4.2.3 Requirements for building documentation
  - 4.3 Getting the source code
  - 4.4 Configuring `make`
    - 4.4.1 Build modes
    - 4.4.2 Running `autogen.sh`
    - 4.4.3 Running `configure`
      - Configuration options
      - Checking build dependencies
      - Configuring target directories
  - 4.5 Compiling LilyPond
    - 4.5.1 Using `make`
    - 4.5.2 Saving time with the `-j` option
    - 4.5.3 Useful `make` variables
  - 4.6 Post-compilation options
    - 4.6.1 Installing LilyPond from a local build
    - 4.6.2 Generating documentation
      - Documentation editor’s edit/compile cycle
      - Building documentation
      - Building a single document
      - Saving time with `CPU_COUNT`
      - Installing documentation
      - Building documentation without compiling
    - 4.6.3 Testing LilyPond binary
  - 4.7 Problems
  - 4.8 Concurrent stable and development versions
    - 4.8.1 Replacing the notation fonts in development versions
  - 4.9 Build system
- 5 Documentation work
  - 5.1 Introduction to documentation work
  - 5.2 `\version` in documentation files
  - 5.3 Documentation suggestions
  - 5.4 Texinfo introduction and usage policy
    - 5.4.1 Texinfo introduction
    - 5.4.2 Documentation files
    - 5.4.3 Sectioning commands
    - 5.4.4 Menus
    - 5.4.5 LilyPond formatting
    - 5.4.6 Text formatting
    - 5.4.7 Syntax survey
      - Comments
      - Cross-references
      - External links
      - Fixed-width font
      - Indexing
      - Lists
      - Special characters
      - Miscellany
    - 5.4.8 Other text concerns
  - 5.5 Documentation policy
    - 5.5.1 Books
    - 5.5.2 Section organization
    - 5.5.3 Checking cross-references
    - 5.5.4 General writing
    - 5.5.5 Technical writing style
  - 5.6 Tips for writing documentation
    - 5.6.1 Working on subsections
    - 5.6.2 Searching
  - 5.7 Scripts to ease documentation work
    - 5.7.1 Building only one section of the documentation
    - 5.7.2 Updating documentation with `convert-ly`
  - 5.8 Documentation strings in the Internals Reference
  - 5.9 Translating the documentation
    - 5.9.1 Getting started with documentation translation
      - Translation requirements
      - Which documentation can be translated
      - Starting translation in a new language
    - 5.9.2 Documentation translation details
      - Files to be translated
      - Translating the website and other Texinfo documentation
      - Adding a Texinfo manual
    - 5.9.3 Documentation translation maintenance
      - Check state of translation
      - Updating documentation translation
      - Updating translation committishes
      - Maintaining without updating translations
    - 5.9.4 Technical background
- 6 Website work
  - 6.1 Introduction to website work
  - 6.2 Uploading website
  - 6.3 Debugging website and docs locally
  - 6.4 Translating the website
- 7 LilyPond Wiki work
  - 7.1 Introduction to the LilyPond Wiki
  - 7.2 Adding and editing snippets
  - 7.3 Snippet administration
  - 7.4 The `makelsr.pl` script
  - 7.5 LilyPond Wiki to Git
- 8 Issues
  - 8.1 Introduction to issues
  - 8.2 Triaging bugs
  - 8.3 Issue classification
  - 8.4 Adding issues to the tracker
- 9 Regression tests
  - 9.1 Introduction to regression tests
  - 9.2 Precompiled regression tests
  - 9.3 Compiling regression tests
  - 9.4 Regtest comparison
  - 9.5 Pixel-based regtest comparison
  - 9.6 Finding the cause of a regression
  - 9.7 MusicXML tests
- 10 Programming work
  - 10.1 Overview of LilyPond architecture
  - 10.2 LilyPond programming languages
  - 10.3 Programming without compiling
  - 10.4 Finding functions
  - 10.5 Code style
    - 10.5.1 Languages
    - 10.5.2 Filenames
    - 10.5.3 Code formatting
    - 10.5.4 Naming Conventions
    - 10.5.5 Broken code
    - 10.5.6 Code comments
    - 10.5.7 Handling errors
    - 10.5.8 Localization
  - 10.6 Warnings, Errors, Progress and Debug Output
  - 10.7 Debugging LilyPond
    - 10.7.1 Debugging overview
    - 10.7.2 Debugging C++ code
    - 10.7.3 Debugging Scheme code
    - 10.7.4 Debugging scoring algorithms
    - 10.7.5 Debugging skylines
  - 10.8 Tracing object relationships
  - 10.9 Tracing processing time
  - 10.10 Adding or modifying features
    - 10.10.1 Write the code
    - 10.10.2 Write regression tests
    - 10.10.3 Write convert-ly rule
    - 10.10.4 Automatically update documentation
    - 10.10.5 Manually update documentation
    - 10.10.6 Edit changes.tely
    - 10.10.7 Verify successful build
    - 10.10.8 Verify regression tests
    - 10.10.9 Post patch for comments
    - 10.10.10 Push patch
    - 10.10.11 Closing the issues
  - 10.11 Iterator tutorial
  - 10.12 Engraver tutorial
    - 10.12.1 Useful methods for information processing
    - 10.12.2 Translation process
    - 10.12.3 Listening to music events
    - 10.12.4 Acknowledging grobs
    - 10.12.5 Engraver declaration/documentation
  - 10.13 Callback tutorial
  - 10.14 Understanding pure properties
    - 10.14.1 Purity in LilyPond
    - 10.14.2 Writing a pure function
    - 10.14.3 How purity is defined and stored
    - 10.14.4 Where purity is used
    - 10.14.5 Case studies
    - 10.14.6 Debugging tips
  - 10.15 LilyPond scoping
  - 10.16 Regular expressions
  - 10.17 Scheme->C interface
    - 10.17.1 Comparison
    - 10.17.2 Conversion
  - 10.18 Garbage collection for dummies
  - 10.19 LilyPond miscellany
    - 10.19.1 Spacing algorithms
    - 10.19.2 Info from Han-Wen email
    - 10.19.3 Music functions and Guile debugging
    - 10.19.4 Articulations on EventChord
- 11 Release work
  - 11.1 Development phases
  - 11.2 Release checklist
  - 11.3 Major release checklist
- 12 Modifying the Emmentaler font
  - 12.1 Overview of the Emmentaler font
  - 12.2 Font creation tools
  - 12.3 Adding a new font section
  - 12.4 Adding a new glyph
  - 12.5 Building the changed font
  - 12.6 METAFONT formatting rules
- 13 Administrative policies
  - 13.1 LilyPond is GNU Software
  - 13.2 Environment variables
  - 13.3 Performing yearly copyright update (“grand-replace”)
- Appendix A GNU Free Documentation License