AlphaXmas: The Final Project

Creative Programming And Computing 2020-2021

Graphical Part: a 3D tree growing model
Music generation: based on LSTM neural network model, learning from ABC notation dataset
Our motivation: to discover the computational creativity's to generate music composition and graphics, to present the audience with a creative experience
The final product is available online alphaxmas.bubblefish.studio

Table of Content

1. Overview of the project

2. Background

3. Music Generation Model

4. Graphical Generation Part

5. System Design and Implementation

6. Human Computer Interaction

7. Conclusion and Discussion

1 Overview

an animation of tree growing

on a 3D canvas

computer generated music,

played by the sound font of a music box

+

What is the structure of Picea abies
How to describe the rewriting rules
How to do the rendering of animation in Processing
How to do the animation of plant development in a computational efficient way

2 Background

LSTM neural network model, how to use the technique to generate sequence of music notes, in the field of artificial intelligence
Plant development/animation study, related with formal languages, L-systems and computer graphics/vision
Especially parametric L-system and Turtle Interpretation on 3D canvas
System design, putting them together

plant development: description, iteration, animation

music generator: design, training

3 Music Generation Model

In the music generation part, we explored deep learning technique because hand crafted features are hard to use.
LSTM model is possible to deal with sequence of music notation(pitches and durations), and learn from data directly.
The dataset we used is a historical set with over 1k Christmas carols.
We did preprocessing(filtering, transposing, cleaning) and analysing it with Music21.
We did a vocabulary mapping into a vector space of 34 unique pitches and 29 durations.

3 Music Generation Model

loss optimization Adam: to obtain the minimum loss.
for each iteration, the sparse categorical cross-entropy is used.
we obtained an optimal model after around 95 iterations.

4 Graphical Part

parametric L-system
turtle interpretation
3D model computation

Picea abies

4.1 Graphical Part: Parametric L-system

$$L = \langle G, \omega, P \rangle$$

Axiom:

$$\omega = A_4$$

Production rules:

$$A_n\rightarrow B_{n-1}$$

$$B_n \rightarrow A_n C_n A_n$$

Rewriting steps:

$$A_4$$

$$B_3$$

$$A_3 C_3 A_3$$

$$B_2 C_3 B_2$$

$$\dots$$

an example of L-system

4.2 Graphical Part: Turtle Interpretation

turtle commands

turtle orientation axes

L-system for our tree

4.3 Graphical Part: 3D Model

Approximation using segments:

a set of thick brown segments (trunk, branches)
a set of thin green segments (leaves)

a snapshot of the model

Turtle state $T_i$:

Position $P_i = (x_i, y_i, z_i)$
Orientation (quaternion $q_i$)

Segment: $\langle V_a, V_b \rangle$

Stroke: $\langle P_i, P_{i+1} \rangle \quad P_{i+1} = P_i + d \hat{H}$

Rotation: $q_{j+1} = r \cdot q_j$

5 System Design and Implementation

Music model preparation:

Final application:

Dataset processing and music playback:

Quaternion

Turtle

Tree

Grammar

p5.Geometry

5 Regarding the Results

from a draft to concept design
we made the demo video on hackathon day
we finished the final product

alphaxmas.bubblefish.studio

demo video at first

6 Interaction

3d canvas iteration: right now we have different view point, controlled by mouse movement.
future extension: it is possible to add more interactions based on the context

7 Conclusion

https://arxiv.org/abs/1412.3191, Bach in 2014: Music Composition with RNN

https://www.csh.rit.edu/~aidan/portfolio/3DLSystems.shtml

http://paulbourke.net/fractals/lsys/

https://www.sidefx.com/docs/houdini/nodes/sop/lsystem.html

https://forums.odforce.net/topic/30160-houdini-l-system-based-procedural-forest-and-plants-generator/

http://www.gib.uni-tuebingen.de/image/ausgaben-3?function=fnArticle&showArticle=105

https://anatolijzykov.artstation.com/projects/GKNn4

Thanks for watching!

a tree animation in a parametric L-system model
a Christmas style music melody
listening impression from the audience
future works can be included both from the graphical part and the music part

Reference

Updates 5 new features

Human computer interaction: facial recognition
Is it a good music melody?
If the music fits the standards of a good music melody, a star would appear
Enhanced connection between the generated music and the tree graphics: the blinking light object on the branch
Every tree is unique now, angle and length variability of the growing branch
A loading/start page to avoid different "auto-play" behaviour on browsers

Human Computer Interaction: by facial recognition

keras-posenet

A Good Melody or not

melody has a cadence
(II-V-I, II-I or VII-I)

Enhanced Connection with Music: the decorative lights

Lights rotate colors every time a note is played

Every Tree is Unique