How does sound travel? How does it relate to pitch, melody and everything music? Can we visualize and interact with it to better understand how music works?

The book Code of Music by Luisa Pereira answers these questions to explain the basics of music theory and then gradually starts diving into interactively teaching how anyone can create their own music with code!!

I worked with her for my Google Summer of Code to make an interactive web and pdf version of the book, along with fellow ITP student Elias Jarzombek, who was also a web developer on the team.

Before we began working on the book style and coding the interactives, we had a lot of previous work to do:

1. Build an infrastructure that would house and structure the book contents.
2. With the book containing a lot of interactives and media, think of a way to optimise the performance and make it light.
3. Build an automated workflow that would make deploying and content creation streamlined.
4. Create a style guide and generic reusable functions.
5. Finally, begin working on placing the book contents and building the interactives.

• Infrastructure

We chose to go with using Magicbook as a platform as it provides a way to create web and pdf versions of  the book from the same source files. Magicbook is an open source project funded by New York University's Interactive Telecommunications Program, where I am currently a student. Apart from providing web and pdf builds, Magicbook has a quick way to set the book structure with Chapters and subchapters, and it generates a  navigation menu without additional styling. It also gives the flexibility to add content in both html and markdown.

The first version of the Code of Music was written in Jekyll which had few issues for our use case. One that was of major concern to us was performance with having multiple iframes for the interactives. We wanted to have a way to use inline canvases for our interactives. So naturally the steps that followed were -

1. Port the existing book from Jekyll to magic book.

2. Test out the libraries that were essential to the project and figure out how they could be bundled with Magicbook.

3. P5.js is a very crucial library for the project as all of our interactives would be built using that. To be able to use the p5 javascript files from other folders to our main content markdown files, Elias built out a separate plugin so that magicbook could render our interactives as a separate canvas.

• Automation

  We spent a lot of time thinking of the best way to automate and streamline our workflows. After a lot of to and fro, we reached a solution(still can do with some more polishing) that we seemed happy with.

  What we needed to automate was -

  1. Porting the content from notion to the website repository.

  Luisa writes all content for the book in notion, as it is a great tool for sharing, getting comments and collaboration. This content would have very frequent updates. It could have been a pretty time consuming process to copy paste the notion content to our repo each time an update happened.

  We wanted to automate this.

  After exploring existing GitHub actions built for this we decided it would be best if we started by building our own NodeJS script. The script looked for all the chapter related content on notion, via the notion API. Based on the type of the content returned by the notion API (heading, bulleted lists, links etc.), we converted the notion content to markdown content and pushed the updated content to the GitHub repo. This script was then converted to a GitHub action which could be future used by our workflows.

  <aside> 💡 The code for which can be found here

  </aside>

  1. Deployments to staging and production

  For each time a commit happens on the master branch ****we build out GitHub ****workflows to deploy to our staging and production.

• Interactives

  We used p5js to build all our interactives which were supported by other libraries like - ml5js, tone js and essentiajs.

  The book is planned to contain 2 types of interactives,

  1. The reader understands musical concepts(melody, pitch etc) by interacting with a sound source(pre recorded or mic input)
  2. The reader understands how code affects the parameters(melody, pitch etc) by playing with the code and seeing the change in real time

  We began by working on the first kind of interactives first.

  • Our first challenge was to find suitable libraries that would take sound from a reader’s mic and determine parameters like pitch,amplitude.
  • Our second challenge was to design the interactions in a fun and user-friendly way.
  • Our third challenge was to make sure the interactions were fluid. So that the idea of visualising sound was conveyed well.

  Luisa made sure we had a smooth workflow, from designing the UX of the interactions, to including the assets. Once the logic for the interaction was coded, she would playtest it and we would iterate on her feedback. The interactions are still a work in progress, where we still need to refine the styling, add all the other planned interactives.

  But it was a fun process to build out a standardised workflow of building and integrating the interactives to the book.

  <aside> 💡 The repository for Code Of Music book can be found **here and a deployed version of the book here**

  </aside>

  <aside> 💡 With link to my specific commits found here

  </aside>