1 - Cerimes

Music and Text Generation
“in the style of”
François Pachet
SONY CSL & LIP6, UPMC
Constraints and composition
Constraints and text writing
Palindromes:
- « A man, a plan, a canal: Panama »
- Georges Perec’s palindromes (1,247 words)
Lipogrames
- G. Perec “La Disparition” without voyel “e”
- “Les revenentes (Perec, texte)”, with only voyel ”e”
« Telles des chèvres en détresse, sept
Mercédès-Benz vertes, les fenêtres crêpées
de reps grège, descendent lentement West
end Street et prennent sénestrement
Temple Street vers les vertes venelles
semées de hêtres et de frênes près
desqelles se dresse, svelte et empesé en
même temps, l'Evêché d'Exeter.
Creativity often arises from playing
with styles
Ghedini et al. The Flow Machines project. Ijcai 2013 and AAAI 2013 best video award
Imitative Sequence Generation
Given a corpus, i.e. a set of finite sequences
𝐶 = 𝑆1 , 𝑆2 , … , 𝑆𝑛 𝑤ℎ𝑒𝑟𝑒 𝑆𝑖 = 𝑆𝑖1 , 𝑆𝑖2 , … , 𝑆𝑛𝑘
Generate 1 sequence / the best
Generate all sequences
Generate a representative subset of
sequences (≈ sampling)
That sounds/reads/looks like 𝐶
AND
enforce specific properties:
domain-dependent
user defined
Imitative Sequence Generation
Given a corpus, i.e. a set of finite sequences
𝐶 = 𝑆1 , 𝑆2 , … , 𝑆𝑛 𝑤ℎ𝑒𝑟𝑒 𝑆𝑖 = 𝑆𝑖1 , 𝑆𝑖2 , … , 𝑆𝑛𝑘
Optimization
Generate 1 sequence / the best
Generate all sequences
Generate a representative subset of
sequences (≈ sampling)
That sounds/reads/looks like 𝐶
AND
enforce specific properties:
domain-dependent
user defined
Imitative Sequence Generation
Given a corpus, i.e. a set of finite sequences
𝐶 = 𝑆1 , 𝑆2 , … , 𝑆𝑛 𝑤ℎ𝑒𝑟𝑒 𝑆𝑖 = 𝑆𝑖1 , 𝑆𝑖2 , … , 𝑆𝑛𝑘
Optimization
Generate 1 sequence / the best
Generate all sequences
Generate a representative subset of
sequences (≈ sampling)
Statistical inference
That sounds/reads/looks like 𝐶
AND
enforce specific properties:
domain-dependent
user defined
Imitative Sequence Generation
Given a corpus, i.e. a set of finite sequences
𝐶 = 𝑆1 , 𝑆2 , … , 𝑆𝑛 𝑤ℎ𝑒𝑟𝑒 𝑆𝑖 = 𝑆𝑖1 , 𝑆𝑖2 , … , 𝑆𝑛𝑘
Optimization
Generate 1 sequence / the best
Generate all sequences
Generate a representative subset of
sequences (≈ sampling)
Statistical inference
That sounds/reads/looks like 𝐶
AND
enforce specific properties:
domain-dependent
user defined
CSP & Global constraints
Imitative Sequence Generation
Given a corpus, i.e. a set of finite sequences
𝐶 = 𝑆1 , 𝑆2 , … , 𝑆𝑛 𝑤ℎ𝑒𝑟𝑒 𝑆𝑖 = 𝑆𝑖1 , 𝑆𝑖2 , … , 𝑆𝑛𝑘
Optimization
Generate 1 sequence / the best
Generate all sequences
Generate a representative subset of
sequences (≈ sampling)
Statistical inference
Information geometry
That sounds/reads/looks like 𝐶
AND
enforce specific properties:
domain-dependent
user defined
CSP & Global constraints
Style and Markov chains
Markov Hypothesis
𝑃 𝑠𝑖 𝑠1 , 𝑠2 , . . , 𝑠𝑖−1 ) = 𝑃 𝑠𝑖 𝑠𝑖−1 )
Random walk
Generating Markov Sequences
Order 1
𝑋1
𝑋2
𝑃 𝑠𝑖 𝑠1 , 𝑠2 , . . , 𝑠𝑖−1 ) = 𝑃 𝑠𝑖 𝑠𝑖−1 )
Order 2
𝑋1
𝑋2
𝑋3 𝑃 𝑠𝑖 𝑠1 , 𝑠2 , . . , 𝑠𝑖−1 ) = 𝑃 𝑠𝑖 𝑠𝑖−1 , 𝑠𝑖−2 )
Variable order with max bound
Random walk:
generate X1 with prior P(X1 ),
then X1 with P(X2 X1
then find the longest prefix 𝑘 ≤ 𝑚𝑎𝑥 for which
there are at least 𝑝 continuations (𝑝 ≥ 1) and draw
P(Xn Xn−k , Xn−k+1 , ⋯ , Xn−1
Markov and Music Improvisation:
Continuator
Interactive Markov chains for stylistic imitation
Alan Silva
Bernard Lubat
A Veenendaal
Pachet, F. The Continuator: Musical Interaction with Style J. of New Music Research, 2003, best paper award
Pachet, F. Music Interaction With Style in SIGGRAPH 2003 Abstracts and Applications, San Diego, 2003
Continuator
Continuator with children
Addessi, A.-R. and Pachet, F. Experiments with a Musical Machine: Musical Style Replication in
3/5 year old Children. British J. of Music Education, 22(1):21-46 March 2005
2 days later, the child
invents a new style,
which sounds like Bach
arpeggios …
Arranging
Style?
Accompaniment
Composition
Improvisation
Problem: control is incompatible with
Markov
=
𝑃 𝑠𝑖 𝑠1 , 𝑠2 , . . , 𝑠𝑖−1 ) ≠ 𝑃 𝑠𝑖 𝑠𝑖−1 )
Because of
Long-range correlations
Global constraints
control ?
Unary, binary, nary constraints (music « rules »)
Max order (avoid plaggiarism)
Meter (essential !)
Cardinality (grains of salts)
Alldiff (ensure diversity)
Spreading and distributions (natural properties, e.g. 1/f)
Etc.
.. impossible with random walk in
Markov chains
Unary, binary, nary constraints (music « rules »)
Max order (avoid plaggiarism)
Meter (essential !)
Cardinality (grains of salts)
Alldiff (ensure diversity)
Spreading and distributions (natural properties, e.g. 1/f)
Etc.
Constrained Markov sequences:
a new class of problems
General Solution for optimization problems:
Pachet, F. and Roy, P. Markov constraints: steerable generation of Markov sequences,
Constraints, 2011.
Unary constraints solved in linear time:
Pachet, Roy & Barbieri, Finite-Length Markov Processes with Constraints, IJCAI 2011
Meter in pseudo-polynomial time:
Roy, P. and Pachet, F. Enforcing Meter in Finite-Length Markov Sequences. AAAI, 2013
MaxOrder (= enforcing novelty) in linear time
Papadopoulos, A., Roy, P., Pachet, F. Avoiding Plagiarism in Markov Sequence
Generation, AAAI, 2014
http://www.flow-machines.com/maxOrder
Distribution (= spectrum, 1/f)
Pachet, F. Roy, P. Sakellariou, Generating 1/f noise sequences as a CSP, IJCAI, 2015
Virtuoso phrase example:
Stefano di Battista, « Night in Tunisia »
The Physiological Perspective on
Virtuosity
– Strong human motor and perception limits (see
extreme drumming)
– Virtuoso = 10,000 hours of practice (Sloboda et al.)
– Virtuosos dont make mistakes, because they
suppress the slow monitoring functions of the brain:
Virtuosos avoid the speed traps of their pre-frontal
cortices1
In other words: they don’t think
1
Justin London, The Psychology and Neurobiology of Musical Virtuosity, Whitehead Lecture in
"Cognition, Computation, and Culture," given at Goldsmiths College, U. London, 2010.
The AI View on Virtuosity
• Virtuoso do extraordinary things (super
experts)
• Compilation of know-how in the body is a way
to « solve problems »
• From the viewpoint of Markov chains:
Well-defined, apparently hard problem
« Virtuosos are NP-hard problem solvers »
Pachet, F. Bebop Virtuosity Explained McCormack & d'Inverno, Eds. Computers and Creativity,
Springer, 2012
Unary constraints can be solved in
real-time
If one considers only:
– Unary (1 variable) constraints at any position in
the sequence, or
– n-ary between n consecutive states (n < order)
Then ∃ a Markov chain M’ s.t.
– M’ generate only the sequences satisfying the
constraints
– M’ and M are statistically equivalent
=> virtuosity problem solved
Pachet, Roy & Barbieri, Finite-Length Markov Processes with Constraints, IJCAI 2011
Example
Generate from this example
Generate all 4-notes melodies ending with D with their
correct probabilities (sampling)
– Reformulation as CSP with 4 variables
– Arc-consistency of the network with Markov constraint
– Retro-normalisation
(was ½
1/6
1/3)
Papadopoulos, Pachet, Roy, Sakellariou, Exact sampling for regular and Markov constraint with belief propagation, submitted
Virtuoso, with Mark d’Inverno (U. of London)
Markov and meter
General Solution for optimization problems:
Pachet, F. and Roy, P. Markov constraints: steerable generation of Markov sequences,
Constraints, 2011.
Unary constraints solved in linear time:
Pachet, Roy & Barbieri, Finite-Length Markov Processes with Constraints, IJCAI 2011
Meter in pseudo-polynomial time:
Roy, P. and Pachet, F. Enforcing Meter in Finite-Length Markov Sequences. AAAI, 2013
MaxOrder (= enforcing novelty) in linear time
Papadopoulos, A., Roy, P., Pachet, F. Avoiding Plagiarism in Markov Sequence
Generation, AAAI, 2014,
http://www.flow-machines.com/maxOrder
Distribution (= spectrum, 1/f)
Pachet, F. Roy, P. Sakellariou, Generating 1/f noise sequences as a CSP, IJCAI, 2015
Meter
• In the style of M. Legrand
• Total duration = 8-bars
• No note spanning bar lines
𝑛
𝑖=1 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛(𝑛𝑜𝑡𝑒𝑖 )
𝑛𝑜𝑡𝑒𝐾 is 𝐾
𝑖=1 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛(𝑛𝑜𝑡𝑒𝑖 )
• Total duration is
• Duration until
Impossible to explore all chains
…
• For n=23, there are 13 millions paths
• Exponential growth with sequence length
Additive number theory
• Counting integer points:
– Polytopes
– Sumsets ℎ𝐴 = 𝑎1 + 𝑎2 + ⋯ + 𝑎ℎ | 𝑎𝑖 ∈ 𝐴
– Naively: ℎ(𝐴) ≈ ( 𝐴 /2)ℎ
• Khovanskii theorem: the number of integer
points in a convex polytope is a polynomial:
– ℎ(𝐴) = 𝑃(ℎ) with P polynomial (for h big enough)
– Degree of P is < 𝐴
Khovanskii, A. 1992. Newton polyhedron, Hilbert polynomial, and sums of finite sets.
Functional Analysis and Its Applications 26(4):276–281
After 4 steps
• N = 4 => more than 130 paths
• But only 3 unique path lengths: 4, 7 and 10
We explore the set of all path lengths
• n=23, 13 millions paths
• Only 9 unique lengths !
Capturing the style of Ray d’Inverno
Original recording of Ray’s comping on Girl From Ipanema
Constraint: Giant Steps score
Basic rendering
Ray’s reaction
Hi Francois, It is fantastic. Well done.
It is very realistic - you can even hear the wrong notes in some of the
chords !
Keep up the good work.
Cheers,
RealRay
Pachet, F. and Roy, P. Beyond minus ones: Virtual Band Siggraph talk and demo, Los Angeles, 2012
Giant Steps by Wagner
Capture the style of Take 6!
One of the best a capella jazz
groups. 10 Grammy awards
Rich harmonisations, very
recognisable
Very hard to imitate, even to
transcribe!
Reaction of composers: Ivan Lins’ The Island
Grammy-winning Brazilian songwriter. His hit "Love Dance" is one of the most rerecorded songs in musical history (Wikipedia)
The Island harmonized by Take 6
Rio, 2013
Pachet & Roy, Non-Conformant Harmonization: The Real Book in the style of Take 6, Int. Conf. on Comput. Creativity, 2014.
Beyond random walk:
principled generation
General Solution for optimization problems:
Pachet, F. and Roy, P. Markov constraints: steerable generation of Markov sequences,
Constraints, 2011.
Unary constraints in linear time:
Pachet, Roy & Barbieri, Finite-Length Markov Processes with Constraints, IJCAI 2011
Meter in pseudo-polynomial time:
Roy, P. and Pachet, F. Enforcing Meter in Finite-Length Markov Sequences. AAAI, 2013
MaxOrder (= enforcing novelty) in linear time
Papadopoulos, A., Roy, P., Pachet, F. Avoiding Plagiarism in Markov Sequence
Generation, AAAI, 2014,
http://www.flow-machines.com/maxOrder
Distribution (= spectrum, 1/f)
Pachet, F. Roy, P. Sakellariou, Generating 1/f noise sequences as a CSP, IJCAI, 2015
The Max Order problem
Max Order
Def: The max order of a generated sequence is the
maximum length of replication in the original
corpus.
Example: an order 1 Markov sequence with max
order 7
The MaxOrder automaton
L 𝑀𝑎𝑥𝑜 𝑚𝑎𝑥
=𝐿 𝑀 ∩
𝐿 𝐴 𝑛𝑜𝑔𝑜𝑜𝑑𝑖
𝑖=1, 𝑐𝑜𝑟𝑝𝑢𝑠 −𝑚𝑎𝑥
Our contribution:
• Build this automaton quickly
• Naive approach not polynomial, Polynomial construction algorithm inspired by 1, 2
• Automaton is fed to the regular constraint to create ALL length-L sequences
Villeneuve, D., and Desaulniers, G. 2005. The shortest path problem with forbidden paths. European Journal of Operational
Research 165(1):97–107.
Aho, A. V., and Corasick, M. J. 1975. Efficient string matching: An aid to bibliographic search. CACM 18(6):333–340.
Gilles Pesant: A Regular Language Membership Constraint for Finite Sequences of Variables. CP 2004: 482-495
The MaxOrder automaton
Papadopoulos, Roy & Pachet, Generating non-plagiaristic Markov sequences with max order Sampling,
Creativity and Universality in Language, Degli Esposti, Altmann, Pachet Eds, Springer, Morphogenesis series,
2015
Papadopoulos, Roy & Pachet, Avoiding Plagiarism in Markov Sequence Generation, AAAI, 2014
Same with Leadsheet generation
Pachet, F. and Roy, P. Imitative Leadsheet Generation with User Constraints, ECAI 2014
0,9
MIN ORDER = 3 & MAX ORDER = 10
0,8
Distribution of chunk size
0,7
Order min 1
0,6
Order min 2
Order min 3
0,5
Order min 4
0,4
Junk
Sweet spot
Plagiarism
Order min 3
max 10
0,3
0,2
0,1
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
MaxOrder Constraint
Leadsheet Generation
LSD: http://lsdb.flow-machines.com
Pachet et al, A Comprehensive Online
Database of Machine-Readable Lead
Sheets for Jazz Standards, ISMIR 2013
Pachet & Roy, Imitative Leadsheet
Generation with User Constraints,
ECAI 2014
www.flow-machines.com/leadsheetgeneration
Flow Composer
There is a huge difference between a
SCORE and actual MUSIC
Without production
Original (produced)
Without production
Original (produced)
Without production
Original (produced)
Music Synchronization
(with M. Marchini)
Revisiting Ode to Joy (Beethoven)
From Prayer in C (Lilly Wood & The Prick)
From Penny Lane (the Beatles)
From Chi Mai (Ennio Morricone )
Marchini, Pachet, Roy, Synchronization Constraints: An Audio-content Based Method for
Generating Multi-Instrumental Music, submitted Ismir 2015
Brazil guitar style capture
Style Capture in Brazil
Listening to Generation
Generation examples
Original played by various musicians
Automatic generation on arbitrary leadsheets
Insensatez by Geber Ramalho
Blue in Green
ReChord generation
Arrastao by Giordano Cabral
Canto Triste (Edu Lobo)
Pop chords by Mathieu Ramona
Michelle
Friends to go by Benoit Carré
Ramona, Pachet Groove preserving concatenative synthesis, submitted
All of me
Benoit Carré Case Study
French Pop Artist
Original version by Benoit Carré
Version by Enzo Enzo, popular
artist in France
Michelle
Original guitar accompaniment by Benoit Carré
Blue in Green
ReChord generation
Markov Constraint applied to text
« Yesterday » by the Beatles
Barbieri, Pachet, Roy & Degli Esposti, Markov Constraints for Text Generation with Style, ECAI
2012
Yesterday … by Bob Dylan
Innocence of a story I could leave today
When I go down in my hands and pray
She knocked upon it anyway
Paradise in the dark side of love it is a sin
And I am getting weary looking in
Their promises of paradise
Now I want to know you would be spared this day
Wind is blowing in the light in your alleyway
Innocence in the wind it whispers to the day
Out the door but I could leave today
She knocked upon it anyway
Khatchatourov, Pachet et al. Action Identity In Style Replication Systems , submitted to Music
Perception
Yesterday by the Beach Boys
Anyone to call after night and start today
Back home let watch the long time I pray
Here we surround you anyway
Heavenly skies and rock like that in my mind hey
Walk a summer summer summer day
You realize a deity
At a time for a change of the while I say
Guys I wonder love I have took a runaway
Anyone to me yeah I over heard you say
Moved her so you will love like today
Here we surround you anyway
Yesterday by ACDC
Fortunate if you wanna take you out to play
Stage I'm big and I want to say
I get enough and I can play
Satellite blues yeah yeah yeah yeah yeah you shook me
Toss off buddy she's gotta see
A fireball in the back once
Stage I'm in the high I said it's way
Toss off buddy she's got it and I'm hey
Fortunate if you ain't nothin I can play
Played all the time you just keep away
Yesterday by Johnny Cash
Theater of your liquor I don't take pay
Gone in the clouds how to live this way
Hit him again and when my way
Theater of your plans I'm down and tell me
Played the boogie in the saddle he
A wanderer a wandering
Play for you and I know what's the way they
Played the boogie in the harp with the key the way
Honeycomb and live in the water and bread they
Play in the sun and I rode away
An interesting and complex global
property: Palindromes
• Palindromes are extreme display of human
creativity / virtuosity
• DOC NOTE: I DISSENT. A FAST NEVER
PREVENTS A FATNESS. I DIET ON COD
(Peter Hilton: “one full sleepless night”)
Palindrome Generation
• Goal: construct palindromic sequences from a corpus
of N-grams (e.g., google n-grams, text, any sequences,
etc.)
• Major difficulty:
– Two levels are inter-related: LETTERS and WORDS
• No algorithm to solve this problem
–
–
–
–
–
–
Brute force
Stochastic, HMM, Monte-Carlo, Metropolis
Double recursion
Automata
Graphs
Constraints
The Palindrome Graph
(with A. Papadopoulos and J.-C. Régin)
CONJUNCTION graph: Gf x Gb
= Tensor product AND encodes the “same character” relation
Papadopoulos, Roy, Régin, and Pachet, Generating all Possible Palindromes from Ngram Corpora, IJCAI 2015
Palindromes: Examples
•
•
•
•
•
•
•
•
•
•
•
‘Evil on an olive’,
•
‘To lay a lot’,
‘Born a man, rob’,
‘Till I kill it’,
‘God all I had, I hid: a hill a dog’,
‘God, a sin: a man is a dog’,
‘Sworn in us at a sun in rows’
‘Drawn in war, died. I set a gate side, I,
drawn in ward’
‘Never a way. By a war even’
‘Evil as a witness is sent, I was a dog;
God as a witness is sent, I was alive’
‘Et on a là, la baraba, là, la note.’
+ very very long ones (80,000 words)
Other Games on Words
• Ambiphrases (each direction in a different
language)
– El, a Roma se dedica -> Acide de sa morale
– No delay -> Y a le don
– Âme de Roy a le don -> No delay or edema
• More to invent…
See interactive demo at Ijcai 2015, Buenos Aires
Conclusion
• Style as a computational object: an ingredient
for creativity enhancing tools,
• New research problems in sampling from
statistical models under global constraints
• Fruitful combinations of ideas from discrete
domain combinatorial optimization, machinelearning and statistical inference
• Industrial potential in entertainment
economy