The department of Petty Hacks and Arcane Knowledge
2020-07-20T05:27:19+00:00
http://danhixon.github.com/
Michael Moser
Goodies
2019-07-24T00:00:00+00:00
http://mosermichael.github.com//all/blog/2019/07/24/goodies
["", "blog"] | <h2 id="goodies">Goodies</h2>
<p class="publish_date">
24 July 2019
</p>
<p>From time to time there are some goodie scripts that I would like to share;
(actually i put up my environment + some more goodies for bash scripting <a href="https://github.com/MoserMichael/myenv">here</a> )
Maybe you might find something interesting within this collection:</p>
<p><a href="https://github.com/MoserMichael/myenv/blob/master/README.md">here</a> are the instructions to set all this all up.</p>
<p>There are also a lot of vim customization <a href="https://github.com/MoserMichael/myenv/blob/master/VIMENV.md">vim customizations & addons</a></p>
<p>anyway, i am now writing less in this blog. i got to think that it is better to set up separate git repos dedicated to some particular subject.
At least we can now make an unlimited number of smaller repos (that wasn’t the case when i started this project),’
My actions in this regard can be seen on my <a href="https://github.com/MoserMichael">github account page</a></p>
<p>Also it seems seems that things have changed a bit since I started this blog (some nine years have passed already).
I am not sure whether it is me who has changed or if it is the world around me that changed: Nowadays I am a bit weary to state my opinions under my own name, I have now the impression that self censorship has become a necessity, I didn’t have that impression back when I started this blog; people somehow seem to have become less tolerant.</p>
<p>If this impression is correct then I would hope this trend would change sooner than later; i think an atmosphere like this is not very conductive to progress, as such.</p>
indoctrination and its opposite
2019-06-26T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2019/06/26/doctr
["", "blogg"] | <hr />
<p class="publish_date">
26 June 2019
</p>
<p>I have been reading a bit on the subject of propaganda language of totalitarian/authoritarian regimes. (here are my <a href="https://github.com/MoserMichael/cstuff/files/3323527/total-language.pdf">notes in pdf format</a> ). And so I came across an interesting encyclopedia article by Charlene Tan that defines the term authoritarian indoctrination, as well as its opposite <a href="https://www.researchgate.net/publication/281224135_Indoctrination">link</a> – I think it is noteworthy that the article does not use the term anti-authoritarian education for the opposite concept.</p>
<p>As I understood the meaning of indoctrination: all incoming information has to be understood exclusively by the subject in terms of the main doctrine; also the school system has to actively instill the doctrine and expunge all doubts and heresies.</p>
<p>Now the problem is that the doctrine is defined as a system of control beliefs, defined as ‘psychologically strong beliefs that are cherished and integral to a person’s life and personal identity’. Now any system of education is based around a set of control beliefs, a lack of such control beliefs would constitute some kind of vacuum. The opposite of authoritarian indoctrination is define as follows: ‘a healthily enculturated person is allowed and encouraged to inquire into and even revise one’s control beliefs if necessary, as well as to interact with and learn from other traditions’ - now my problem is that by actively calling control beliefs into question one would invalidate them as such.</p>
<p>I think that there are other possibilities of a middle ground that are not discussed in this short articles: one could try to go for a historical approach – try to explain how core believes and paradigms came about; how the thinking on a subject changed over time and why it did so. One example: once upon a time people were quite sure all planets were turning around the earth; this was not due to pure ignoreance - they had some good reasons for reaching this conclusion. Still some problems remained with this explanation that could only be explained with an earth at the center of the system. <a href="https://www.khanacademy.org/partner-content/nasa/measuringuniverse/spacemath1/a/the-geocentric-universe">Khan Academy on Geocentrism</a></p>
<p>I think this could possibly allow to present alternatives without invalidating the ‘core beliefs’</p>
<hr />
<p>My notes on the article</p>
<p><a href="https://www.researchgate.net/publication/281224135_Indoctrination">Indoctrination</a> by Charlene Tan</p>
<ul>
<li>Indoctrination – negative term, ‘associated with authoritarian education’. for ‘indoctrination’ you need a system of ‘control beliefs that promote a totalistic ideology’
<ul>
<li>control beliefs := ‘psychologically strong beliefs that are cherished and integral to a person’s life and personal identity They are usually embraced by the person without question and are most resistant … to change.’</li>
<li>‘a totalistic ideology severely limits one’s intellectual horizon by constricting the person to a simplistic and binary ‘We versus You’ worldview. Affectively, such an ideology incites an all-or-nothing emotional alignment through intense affection and loyalty for one’s leaders and fellow group members, and a corresponding hostility and hatred towards those outside the group. What follows in behaviour is a mobilisation of extremist thoughts and destructive emotions to protect one’s ideology, advance its cause and eradicate all obstacles and enemies at all costs.’</li>
</ul>
</li>
</ul>
<p>Features of an indoctrinated person</p>
<ul>
<li><em>indoctrinators strive to isolate their pupils from all factors that challenge their believe systems</em> - they strive to educate a set of core believes that are described as abstract and unchallenged positive value (the capital letter truth)</li>
<li><em>all incoming information has to be in alignment with the doctrine:</em> ‘these control beliefs determine the person’s identity and control one’s entire life, making the person interpret everything through the lens of the control beliefs. These beliefs stubbornly withstand the external challenge and even distort reality by filtering all incoming stimuli and re-interpreting new information in alignment with and support of one’s control beliefs. This is possible because the control beliefs are fortified by a small but carefully implanted and deeply embedded cluster of intertwined beliefs. ‘</li>
<li><em>everything that does not fit the doctrine is rejected</em> ‘an indoctrinated person’s control beliefs screen and censure new inputs that challenge, or are inconsistent with, the existing control beliefs by forming new beliefs to reject them, such as “This thought is from the devil”’
<ul>
<li>“develops intense affection and loyalty for “Us” and a corresponding hostility and hatred towards “Them”. The end result is that the person, feeling privileged to have been ‘chosen’, is obsessed with removing all hindrances – human and otherwise – to fulfill one’s ‘higher calling’ to protect and propagate one’s ideology.’”</li>
</ul>
</li>
</ul>
<p>to make it stick with the pupils: all this must be pushed by ”a community of believers who share the same tradition. An indoctrinatory tradition is a discourse or social process that seeks to entrench control beliefs that promote a totalistic ideology”</p>
<ul>
<li>“there must be a deliberate, systematic and sustained process by the indoctrinator(s) to implant control beliefs that advance a totalistic ideology. “</li>
<li>“By dogmatically insisting that it has the monopoly of the truth, it trumpets its own infallibility, resists genuine learning from other traditions, and censures alternative worldviews. In the process, it fosters closed-mindedness and undermines the basic social conditions for its members to grow and mature in their thought, emotions and actions. “ “incapacitates its members’ development of strong rationality and strong autonomy. Such a tradition may grant its members weak rationality in the sense that they are capable of giving reasons to support their beliefs by assuming the truth of their own tradition. But strong rationality is denied as its members are prohibited from examining or critiquing the tradition itself, considering alternatives, and learning from other traditions.”</li>
</ul>
<p>on the opposite of indoctrination: “ But these (core) beliefs are not held in such an extreme way that they become impervious to doubt. A healthily enculturated person is allowed and encouraged to inquire into and even revise one’s control beliefs if necessary, as well as to interact with and learn from other traditions. The willingness and ability of a normally enculturated person to question one’s own beliefs, consider alternatives and order one’s life autonomously are largely absent in an indoctrinated person; the latter is one who blindly clings on to a totalistic ideology as the person’s cognitive, affective and behavioural growth has been paralysed. “</p>
<hr />
<p>A very interesting exchange that i had on the subject of raising kids: <a href="https://news.ycombinator.com/item?id=22145203">link that includes the discussion</a></p>
<p>RobertRoberts [-]</p>
<p>I found that as a father teaching my children to confidently, and respectfully, challenge me has brought a tremendous amount of peace to the family.</p>
<p>Sometimes you are wrong and you really need help to overcome the emotional baggage that blocks you from admitting it.</p>
<p>Having people in your life that are strong enough to deal with the discomfort of confrontation with the singular goal of peace is more valuable than your pride.</p>
<p>But it also allows the same interaction to be reciprocated when others are wrong in the family and it’s a self supporting system to overcome arrogance, angst and many other negative events and emotions.</p>
<hr />
<p>MichaelMoser123 [-]</p>
<p>how old are your children? At what age did you start to teach them in how to challenge you respectfully ?</p>
<hr />
<p>RobertRoberts 22 days ago [-]</p>
<p>The first moment it happened was when my oldest was about 7 (My oldest is 22 youngest is 13) and she yelled at me for doing something wrong. (I was a lousy parent early on) But she was right and I was wrong, and it hurt really bad (emotionally/pride) but I admitted she was right, but I asked her to speak to me respectfully about when I am wrong.</p>
<p>This took years of patience and still my oldest doesn’t do well with this, but the other kids are fairly ok with it.</p>
<p>It’s not perfect, we all still have some conflict, but the resolutions are swift and emotions don’t linger after a conflict.</p>
<p>It’s taken more time for me to figure out how to treat others well than it has to teach my kids. Once I figured something out, getting them to behave like I am is the easier part.</p>
<hr />
<p>MichaelMoser123 [-]</p>
<p>fascinating. Now what is your immediate reaction upon criticism? Do you respond to the issues raised or do you refrain from reacting? A second question: how do you react when the criticism stops to be respectful?</p>
<hr />
<p>RobertRoberts [-]</p>
<blockquote>
<p>what is your immediate reaction upon criticism?</p>
</blockquote>
<p>Sometimes mad, sometimes I have to pause and take a breath and think about what is happening. Sometimes I know I am wrong and in the heat of the moment I hope someone says something, and I am glad when they do.</p>
<blockquote>
<p>Do you respond to the issues raised or do you refrain from reacting?</p>
</blockquote>
<p>Sometimes I can say sorry immediately, sometimes I argue for a bit, or explain myself and what is going on. Nothing happens in a vacuum, so there’s other events occuring that need to be addressed. (ie other people’s behavior)</p>
<p>Very very rarely do I feel the need to step away and cool off, though my wife has historically preferred this. But the more control over situations I have gained over the years the less she needs to “cool off”.</p>
<p>It’s like if anyone in the room has a cool head and can calm others down and it’s generally agreed in the group that this is the goal, then it only takes that one person’s input and energy to affect the situation.</p>
<blockquote>
<p>how do you react when the criticism stops to be respectful?</p>
</blockquote>
<p>No one is perfect, and I express this. I say “you know I want open criticism, but do it respectfully please”. Or something along those lines. The reason this works though is that I acknowledge when I have failed along the same lines.</p>
<p>I often have to say “sorry, I overreacted, but the main issue is still important to address…”.</p>
<p>I have an expectation of failure in my interactions now. This means if a child attempts to intervene with input or criticism, the first thing that child gets is praise for the attempt. Even if everyone is still angry and yelling, I will address the child’s successful choice in trying to do the right thing. Their heart was in the right place, then second they get a criticism for their poor behavior in their attempt. And then I respond to their specific statement. (usually requiring a change in my own behavior or an apology)</p>
<p>This is all logical and easy to say, but it only works because I’ve been practicing it for a long time. And I openly state (even when failing spectacuraly) that it’s important for all of us to get good at this. And that we will fail repeatedly, but that’s ok.</p>
<p>Keep in mind that the more I do this and the more my family does this, the less situations occur where these kinds of actions are needed. It’s a curious quasi-paradox.</p>
notes on texts that seem to make sense
2019-04-12T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2019/04/12/self-help
["", "blogg"] | <hr />
<p class="publish_date">
12 April 2019
</p>
<p>This page will be silently updated with any new stuff that I am picking up along the way: (I don’t want to put up a new page for every new note, and it is much easier to put it down as a pdf file)</p>
<!--
-=-=-=-=-=-=-=-=-=
I have been consuming quite a lot of TED talks recently; Now TED talks are very feel-good, that's their problem:
ee this Quora discussion [Are the TED Talks a type of church for atheists?](https://www.quora.com/Are-the-TED-Talks-a-type-of-church-for-atheists)
Answer by Bjoern David Paulson:
“... The actual problem comes when the aggregate sum of pop science is billed as something transformative. The problem arises when viewers come for the enthusiasm and reassurance and confuse that with actual understanding. When the hard questions are left behind and all that remains is a comfortable euphoria”
I hope that keeping notes while listening can get you past the feel good effect, it also leads you to searching for more information if a given topic has caught your interest.
//-->
<hr />
<p>Some stuff on the subject of Pixar and storytelling: <a href="https://github.com/MoserMichael/cstuff/releases/download/pixar/pixar_storytelling.pdf">summary as pdf file</a></p>
<hr />
<p>some readings on the subject of self-help, and advise on how to behave. Here is my <a href="https://github.com/MoserMichael/cstuff/files/3071781/attention.pdf">summary as pdf file</a></p>
<hr />
<p>Some interesting articles about the preconceptions and biases of some well known philosophers <a href="https://github.com/MoserMichael/cstuff/releases/download/updsoft/bad_philosophers.pdf">pdf notes here</a></p>
<hr />
<p>An interesting lecture on the Neo-Assyrian empire <a href="https://github.com/MoserMichael/cstuff/releases/download/updsoft/neo-assyrian-empire.pdf">pdf notes here</a></p>
<hr />
<p>Notes on some works on the history of the industrial revolution and why it all started in Britain <a href="https://github.com/MoserMichael/cstuff/releases/download/updjuly/why-did-industrialization-start-in-britain.pdf">pdf notes here</a></p>
<hr />
<p>studied some works on the language of propaganda and indoctrination of totalitarian regimes <a href="https://github.com/MoserMichael/cstuff/files/3323527/total-language.pdf">pdf notes here</a></p>
Michael on Software
2019-03-26T00:00:00+00:00
http://mosermichael.github.com//all/blog/2019/03/26/me-on-software
["", "blog"] | <h2 id="michael-on-software">Michael on Software</h2>
<p class="publish_date">
26 March 2019
</p>
<p>Everyone in this Industry seems to have a playbook</p>
<p>The big incumbents have the goal to stay on top of the heap;
They might follow the advise of the following articles by Joel Spolsky: <a href="https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/">Smart companies try to commoditize their products’ complements</a> ; another one of this is <a href="https://www.joelonsoftware.com/2002/01/06/fire-and-motion/">“fire and motion”</a>
And the upstarts have a playbook of their own; <a href="https://www.youtube.com/watch?time_continue=11&v=rHdS_4GsKmg">disrupt from the low end from position as substitute</a> says Clayton Christiensen</p>
<p>Interesting that each article wants to promote itself as the big Truth; I think that each of them is looking at the same picture from a slightly different perspective.
Another thing is that these models look a bit apologetic, they describe the current state (that’s why the argument is quite convincing); It may by extension describe pattern of coming developments, but it might also fail to do so in the future.</p>
<p>I think that when any player is creating value, he is by extension also creating opportunities for other players. Its the law of unintended consequences that really counts – every action should be examined in terms of opportunities that it creates for other players, sometimes these unintended side effects turn out to be of major importance;
Or: any ecosystem is creating opportunities for a whole range of actors, sometimes the effects af an action is not obvious at first, and the chain of events yields results that contradict the interests of the host/owner of the system.</p>
<p>examples:</p>
<ul>
<li>Sun microsystem developed Java as a tool for <a href="https://en.wikipedia.org/wiki/Java_(software_platform)#History">smart appliances</a>; as a complementing product that would enable SUN to sell more hardware; several years later the major use case turned out to be enterprise software; now Java is a portable system, so that people started to use Java with cheap x86 servers instead of driving the sales of more expensive machines by Sun.</li>
<li>The Facebook API as a tool for third-party integration was used by <a href="https://medium.com/tow-center/the-graph-api-key-points-in-the-facebook-and-cambridge-analytica-debacle-b69fe692d747">Cambridge Analytica</a>, resulting into major fallout for Facebook. Of course users of this social network do get a lot of utility out of it; but the whole story did bring it to their attention that Facbook is not just making money by selling targetted adds; their data is part of the product and is being sold and repackaged for profit; this insight turned into quite a bit of resentment against facebook.</li>
<li>I think very complex social networking sites are becoming hard to govern, because there is this multitude of players who also try to use the system to their own advantage.</li>
<li>Thinking in terms of an ecosystem of different players is harder, but may become necessary when systems start to involve a lot of actors.</li>
</ul>
<p>Another issue is that the industry is becoming a bit more complex and interdependent compared to what we had in the 90ies and turn of the century: this cloud thing is making things quite complex – if big cloud providers are turning enterprise software into ‘complements’; now what is Kubernetes/Docker? Is it a complementing product or will it lead for a push for hybrid clouds? Will hybrid clouds eventually cut into the profit of AWS/Azure/Google Cloud?</p>
<p>This could make a bigger impact if combined with an adoption of ARM based servers, ARM processors need less cooling when compared to x86 processors, so you could possibly pack more systems into a box at a lower purchasing cost. However Mr. Thorvalds says that this is unlikely to happen: x86 became so important because devlopers could use the same platform for development and deployment <a href="https://www.realworldtech.com/forum/?threadid=183440&curpostid=183486">here</a>. This could also change in the future; the <a href="https://www.oreilly.com/library/view/kubernetes-in-action/9781617293726/">Kubernetes book</a> has a fascinating example where they build a playground cluster out of raspberry pis. Now what will happen when these raspberries grow up and if it becomes easier to set up a private cloud rather then paying a cloud provider for its services?</p>
<p>Another aspect is that every player in this game seems to have his own agenda and is pushing into a totally different direction. Also same thing can possibly be both a complement in one context and something completely disruptive if seen in a different context.</p>
<p>My conclusion: one might be analyzing the industry in terms of big players only, similar to the authors cited at the beginning of the article; but then it is easy to miss emerging trends and threats to these players;</p>
<p>Another distrinct possibility is that I am misunderstanding the whole picture; I guess time will tell.</p>
<hr />
<p>Notes on “Strategy letter V” by Joel Spolsky <a href="https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/">link</a></p>
<p>Says the following concept comes from microeconomics – which is less abstract and of more use than macroeconomics:</p>
<ul>
<li>Every product has substitutes and complements;
<ul>
<li>substitute – a product that is similar to but cheaper than the original (chicken substitute of beef)</li>
<li>Complement – a product that goes with another one (gas is a complement of cars)</li>
</ul>
</li>
<li>Rule: demand for substitute increases when price of original product rises; so the substitute becomes more attractive due to low price.</li>
<li>Rule: demand for product increases if the complement becomes cheaper; you get more opportunities with the product and it becomes more attractive.</li>
</ul>
<p>Smart big companies strategy:</p>
<ul>
<li>push down prices of product complements – turn them into commodities “Smart companies try to commoditize their products’ complements” (all to create demand for the core product!)</li>
<li>Example: IBM turned PC add-ins into commodity by documenting the architecture</li>
<li>MS didn’t sell an exclusive MS-DOS license to IBM – MS succeeded in turning the PC into a commodity of MS-DOS.</li>
<li>IBM invests in Open source systems and Linux: they want to commoditize the software environment to create more opportunities for their consulting business (and to drive demand for their own enterprise software and storage systems)</li>
<li>Netscape makes navigator free: they want to sell more http servers and by turning the browser into a commodity they drive demand for http servers (hello apache ;-) And Microsoft make Internet explorer free – so that the web as a commodity would drive demand for Windows installations.</li>
<li>Says Sun microsystems was acting strange/stupid by developing Java – which is portable. Due to this portability it did turn server hardware into a commodity (and server hardware was Sun’s core business)</li>
</ul>
<p>Me: I don’t quite agree with his diagnosis of Sun and java (but I got the advantage of looking back ;-)
I think the initial idea at Sun was to sell more ‘smart devices’ that run java, and that enterprise software was not the initial focus of the java platform, the end effect was something of an unintendended consequence due to the following reasons:</p>
<ul>
<li>Java required a lot of memory and cpu resources; At that time this was too much for the limited hardware of small devices; however a decade later there was android and its apps are based on JVM technology! Now Google somhow got the profit from this technology as it was the company that defined the Android platform; but that is a different story.</li>
<li>Java in the enterprise: Enterprise software turned into a major generator of revenue and here Sun was not a major player; however SUN tried hard to be one in middleware with EJB. This was not a success because EJB was hard to use and the emerging WEB architecture used a different path of development. It was a lost gamble, but that was not quite obvious at the time.</li>
</ul>
<hr />
<p>My notes on <a href="https://www.youtube.com/watch?time_continue=11&v=rHdS_4GsKmg">Clayton Christensen: “Where does Growth come from?”</a></p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/rHdS_4GsKmg" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>Questions:</p>
<ul>
<li>success is hard to sustain; good management causes companies to stumble (!); This creates despair for the author (as expert in management)</li>
<li>growth, were does it come from?</li>
<li>theories of management: many times they don’t help at all! lots of aspects that explained by theory - example: no study of metrics.</li>
</ul>
<p>Observations:</p>
<ul>
<li>says that good metrics would be great for testing theories - like the theory of disruptions? do these metrics exist?</li>
<li>Recessions - hits bottom and need some timer before hiring new people ; for the first six recessions after WWII it took 6 months from end of recession until start of hiring wave. 1991 - 15 months; and 2001 - 39 months; 2008 - six years (!).
Longer gaps are due reasons related to finance (and availability of capital ?) - that is what’s wrong about the economy.</li>
<li>companies are not investing into innovations because of short term focus ; innovations are creating real growth.</li>
</ul>
<p>types of innovations:</p>
<ul>
<li>potential innovations</li>
<li>sustaining innovations</li>
<li>efficiency innovations</li>
<li>disrupting products</li>
</ul>
<p><em>potential innovations</em> - potential products.</p>
<ul>
<li>only 15-25 of all innovations will be financially rewarding - we don’t know which one, so there are a lot of tries;</li>
<li>current marketing studies are crap - studying the customer is the wrong approach; they try to classify customers; and their properties (customer profiles) - which are not significant to the buying decision.</li>
<li>what should be studied is the customer needs (customers not always good at articulating their own needs), a product is there to satisfy a need; always ask ‘what job does this thing solve’? that approach is better at predicting a good product (so it is better to study customer behavior to understand their needs)</li>
</ul>
<p>Research: how can McDonalds improve their milkshakes?
Wrong approach: build a profile of customers and ask them how to improve the product
Right approach: ask existing customers why they use the product? How can we fulfill your needs (the job) better?
Study: when are customers coming? What time? What are they doing with the product - take away or consume on the spot?
Turns out 50% bought it in the morning and drove away ? Asked them ‘what caused you to hire this milkshake’ ‘what was your substitute (focus question)’ - so that we are not hungry on a long drive/commute.
In the afternoon parents buy that for their kids - so they have something to talk over when the kids are consuming the milkshake.</p>
<p>The Customer is wrong unit of analysis - look at the customer problem/situation; and customer needs - to solve the problem that the situation creates.
The task description is static (makes it easy to fix it) - but technology to solve it changes quite a lot;
every Job/customer problem has a functional/emotional/social aspect;</p>
<p>Then ask: what is the experiences are that will fulfill the job done? If these are associate with a brand/business then this would be great.</p>
<p>Ask:</p>
<ul>
<li>how will customers choose us and not the competition?</li>
<li>what can we do that other people can’t ?</li>
<li>How will everyone know about us?</li>
</ul>
<p>Ikea does a great job its because it is organized aroung the ‘job to be done’ (like Ikea);
customers like this approach because the alternative requires more work/effort to search parts for the customer.</p>
<p><em>sustaining innovations</em> - make an existing product better (most number).</p>
<p>don’t create growth because they replace existing things; but are important to sustain existing companies.</p>
<p><em>efficiency innovations</em> - helps to produce more with less; keep us competitive, but reduce jobs and create cash flow;</p>
<p>Examples: Wallmarts, Toyota production process. ;</p>
<p>Abundance and scarcity: some things are scarce - like platinum; have to keep it , some things are abundant - sand, can waste it.
Finance: Capital used to be scarce - now it is abundant (? probably for big players ?) ; also finance now measures success in ratios (metrics) rather than whole numbers (of dollars)
Fractions should allow to performance of companies; (like metrics of ‘return on net assets - RONA’ / ‘internal rate of return - IRR’);
RONA can be gamed by outsourcing rather than doing innovating products;
IRR can be gamed by short range gains (long term investment is not good for it)
Says growth is lost in the pursuit of these metrics. Also companies have a lot of cash on hand;</p>
<p>Efficiency innovations pay off earlier (Pay off in a 6 months - 3 years) so they are prefered to disrupting products (these pay off in 5-10 years)</p>
<p>Japan was growing fast because companies invested in disruptive products; (lots of examples) Since the late eighties they use these metrics - Japan didn’t do a disruptive product ever since.
USA is going the same way; result is that capital is used to create capital.</p>
<p><em>disrupting products</em> - these create growth</p>
<p>Trajectory of improvements that customers are able to utilize (doesn’t change too much).
Trajectory of improvements that companies provide with better proucts - (lot larger than ability to utilize them) -> even not so good products are better than customer expectation.
Example: took a lot of time to utilize the PC;</p>
<p>Says that you can’t unseat a current market leader in their own product (on their field);
But disruptive innovations can - new products that are more accessible and cheaper than existing ones.</p>
<p>Google disrupted advertising
DEC did minicomputers; in good times they sold computers for 200.000$ at 45% margins; Now DEC has disrupted mainframes that sold for a million at >60% margins.
Now the PC didn’t matter for much of the eighties - DEC management cared about their current products, not the low starting PC that would disrupt them.</p>
<p>Innovators dilemma: focus your limited resources on attempt to improve your existing product or care for currently not-relevant but potentially disrupting upstart?</p>
<p>Other examples: Phones :: Blackberry -> Apple -> Samsung (and friends).
Disks :: Big disks -> small ones -> flash.
Cars :: Ford; GM -> Toyota (Corona was affordable to students and disrupted Ford; went after Corolla, etc; now they also sell Lexus cars) -> Hyundai, Chinese are now killing them at the low end.</p>
<p><em>they all disrupt the existing leader from the low ends; the upstarts compete against non-consumption by making consumption more affordable.</em></p>
<p>Q: are there big companies that survived new disrupting products/competing upstarts ?</p>
<p>A: yes, Need to create a new business unit and new processes; like IBM that created new business units for mini computers; later new one for micro computers.</p>
<p>Q: what metrics are good ones for Google?</p>
<p>A: most metrics are concerned with the short term only; useless in the long run, so make up your own metrics.</p>
<p>Q: Google invests in AI: what will be the remaining jobs for humans (if this succeeds) ?</p>
<p>A: applications of AI will probably be pretty complicated; probably this will force us to hire more people.</p>
<p>Q: what is actually the disrupting effect of an innovation ? The technology/the application/the market effect ?</p>
<p>A: Word has a lot of connotations; the business model has to do with it, how it is deployed, and where (not the technology by itself)
Google is good at developing potentially disruptive technologies, but it is not good at creating business models that compete against non-consumption (?)</p>
<p>Q: certain products would not fit in with model of low end disruption: Uber/iPhone/Tesla started from the high end, not the low end ?</p>
<p>A: theory is that Tesla is a sustaining innovation; says that BMW and porsche will give Tesla some heat (or acquire them); Says the transforming innovation will be cheap/small electric cars from China.
Apple: they disrupted the laptop and succeeded. (didn’t compete against blackberry; blackberry didn’t have apps, that’s what killed them); now Apple is being disrupted by cheap android phones.
Uber? taxis are asset intensive (car + medallion) + keep the cars running for the whole day to get a small profit; now Ueber has no assets and variable costs, taxis can’t compete with that; so its also ‘from the low end’ - but in terms of business model</p>
<p>Q: measuring your life in a better way: how can employees do that? Why did you write that?</p>
<p>A: if he analysis personal stories then he sees that (in business terms) the resource allocation process/metrics causes people to pursue a direction that they didn’t want in the first place.
Example: he sees his former friends at Harward reunions; with time fewer people appear at these events because they get divorced and don’t want to talk about it.
The intention was to have happy families says they failed because of wrong metrics (wrong objectives) - on immediate/tangible signs of achievements;
careers get this; kids don’t show this- kids have problems, so they don’t invest in them (its a long term investment)
The book should set this right ‘how to measure your life’</p>
<hr />
<p><a href="https://www.youtube.com/watch?v=DfJ9ydhb53Q">Clayton Christensen On Truth, God and Personal Courage</a></p>
<p>Q: what have you learned since (after many afflictions and illnesses) ?</p>
<p>A: Many attribute the speakers stand due to personal courage, he thinks differently: you do need courage - if you have postponed work or undone work; that leads to fear when faced with danger.
Says he has no fear because he is convinced that he has led the right live according to G-d; so his approach is ‘lets face another problem’</p>
<p>Q: what do you tell people who don’t have faith?</p>
<p>A: atheism is similar to religion because its proposition is an unprovable statement (so it becomes kind of a religion); also both have common features: both Atheism and Religion do have a view of life/code of ethic;</p>
<p>says the conflict/distinction between science and religion is exaggerated - religious people don’t want to face criticism by science, so they don’t talk about Religion.</p>
<p>Says that concept of Greek deities (liake Zeus) is that of being outside of nature; so that science has nothing to do/is incompatible with the will of the deities.</p>
<p>Christiensen says that G-d is not outside of the universe, but within it and that he uses/is governed by natural laws in his actions.
In this view science is not incompatible with Religion, as the truth of science is part of the same realm as religion (or rather deal with the same concepts/have a shared language); as he sees Gods action as being subject to constraints of natural laws and science as studying these natural laws.</p>
<p>Q: who knows that our understanding doesn’t change and will be invalidated in the future.</p>
<p>A: Can’t know that - can hope that our knowledge is somehow converging with truth.</p>
Post-modern C+++
2019-03-21T00:00:00+00:00
http://mosermichael.github.com//all/blog/2019/03/21/cppnotes
["", "blog"] | <h2 id="post-modern-c">Post-modern C+++</h2>
<p class="publish_date">
1 March 2019
</p>
<h3 id="learning-modern-c">learning modern c++</h3>
<p>Here is a very long <a href="https://github.com/MoserMichael/cstuff/releases/download/newcpp/cpp11-scott-meyers-a.pdf">article/tutorial on modern c++</a> (link to pdf file) I wrote it up while commuting on the train, it tries to distill and make sense out several sources on modern C++.</p>
<hr />
<p>some notes on the talk series <a href="https://github.com/MoserMichael/cstuff/releases/download/updjuly/cpp-api-design.pdf">modern c++ api design</a></p>
<hr />
<p>some notes on how to do a <a href="https://github.com/MoserMichael/cstuff/files/3494888/hash-table-cpp.pdf">good hash table</a></p>
lecture notes
2019-03-01T00:00:00+00:00
http://mosermichael.github.com//all/blog/2019/03/01/notes
["", "blog"] | <h2 id="lecture-notes">lecture notes</h2>
<p class="publish_date">
1 March 2019
</p>
<h3 id="some-lecture-notes">some lecture notes</h3>
<p>This page will be silently updated with any new stuff that I am picking up along the way:</p>
<p>-=-=-=-=-=-=-=-=-=</p>
<p>I have been learning the machine learning course by Andrew Ng; <a href="https://www.youtube.com/watch?v=PPLop4L2eGk&list=PLLssT5z_DsK-h9vYZkQkYNWcItqhlRJLN">on youtube</a>
I had to write down my lecture notes, as I have been learning while commuting by train. Here are <a href="https://github.com/MoserMichael/cstuff/releases/download/0%3B01/ml-pdf.pdf">my lecture notes pdf file</a> - link to pdf file.</p>
<hr />
<p>Some stuff that I learned while working on storage systems.
<a href="https://github.com/MoserMichael/cstuff/releases/download/updsoft/storage_notes.pdf">pdf notes here</a> - link to pdf file.</p>
<hr />
<p>Also wrote down my notes on the best book on computer networks: Networking on TCP/IP Illustrated,Volume 1 The Protocols W. Richard Stevens <a href="https://github.com/MoserMichael/cstuff/releases/download/0%3B01/stevens-protocols-notes.pdf">pdf notes here</a> - link to pdf file.</p>
<p>(have to admit it tha I still don’t quite understand ipv6 after all these years, have to learn it some day)</p>
<hr />
<p>Here are my notes on learning about <a href="https://github.com/MoserMichael/cstuff/releases/download/bpf/learning_ebpf.pdf">eBPF</a></p>
<hr />
<p>Here are my notes on lerarning about Kubernetes <a href="https://github.com/MoserMichael/cstuff/releases/download/kbook-upd/kubernetes-book.pdf">pdf notes here</a></p>
<p>and here some notes on learning about <a href="https://github.com/MoserMichael/cstuff/releases/download/k8sop/kubernetes-operators.pdf">Kubernetes operators</a></p>
<p>Some notes on learning <a href="https://github.com/MoserMichael/cstuff/releases/download/lgolang/learning-go-lang.pdf">the go language</a></p>
<p>Here are my notes on learning tmux, a real time saver <a href="https://github.com/MoserMichael/cstuff/releases/download/new-stuff-learned/learning-tmux.pdf">learning tmux</a></p>
<hr />
<p>Some notes on how to do a <a href="https://github.com/MoserMichael/cstuff/files/3494888/hash-table-cpp.pdf">good hash table</a></p>
<hr />
<p>lately I had to do some work with Java, and noticed that a lot of code uses the stream feature of java8; so I had to learn this nice application of monads. Here are <a href="https://github.com/MoserMichael/cstuff/releases/download/new-stuff-learned/java-streams-monad-example.pdf">my notes on the subject</a></p>
<hr />
<p>I came across a nice explanation of how bitcoin and this blockchain thing works; so here are <a href="https://github.com/MoserMichael/cstuff/releases/download/new-stuff-learned/bitcoin-how-it-works.pdf">my notes on the subject</a></p>
<hr />
<p>Also I learned the following tricks in the process:</p>
<ul>
<li>I did the notes on Libre office writer; it turns out that it has a working formula editor with a very nice syntax; Microsoft word is not usable for this purpose, as its formula editor keeps crashing.</li>
<li>Also Libre office has a very nice feature - it can save a document as PDF file (I think MS Office doesn’t know how to do that);</li>
<li>in github you can produce a binary release for a project that contains these pdf files (of course we know that it is a no-no to add binary files to a git archive), the blog in turn can link to these pdf files; this trick saves me the headache of having to reformat everying in markdown and then having to figure out where this mess fails to display properly. This way this blog becomes even more locked down on the github platform (three cheers for github!!!)</li>
<li>(i hope they will not pull another trick here, like the forced transition from redcloth to markdown markup. At least writing most stuff as a pdf will not make a difference when they do)</li>
<li>Some meta observations: most of the notes here are divided into nested lists, this helps me to come up with a more concise representation of complicated syntax structures; In a way this is similar to lisp with nested s-expr syntax; s-expr syntax is more concise than Algol like languages, however it creates problems for programs developed by large teams over a long period of time. Now maybe the whole point of s-expr/lisp is to teach another mode of thinking – by programming in lisp you train your mind to think in more abstract patterns! (it takes a lot to overcome the initial hurdles though – that is another reason why it so far didn’t turn into a mainstream programming language)</li>
</ul>
<hr />
<p>Other note on learning about distributed systems:</p>
<h2 id="map-reduce">Map-reduce</h2>
<p>(i was told tha map-reduce is no longer in use at google, however this paper is quite interesting if you want to learn about storage & distributed systems)</p>
<p>https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf</p>
<p>problem: working on large data sets that are distributed among a huge set of machines: each program needs to deal with node failures and distribution of data and load balancing; that results in lots of duplicated work; Instead use inversion to have a general mechanism to solve these problems in generalized manner.</p>
<p>Simplified model: most computations are two-step</p>
<ul>
<li>perform a map function - it passes over input records and fill output multimap hash table (key can have multiple values)</li>
<li>apply reduce function - it passes over intermediate multimap and combines values with the same key into output value; this output value is passed to a Reduce function as input)</li>
</ul>
<p>Implementation principles</p>
<ul>
<li>works over data stored on google file system</li>
<li>the job is subdivided into set of splits (splits processed in parallel on different machines)</li>
<li>one process applies the map function over a single split</li>
<li>for each split the reduce function is subdivided: reduce can be subdivided into sub sets by applying processor-Index = hash(Key) % NumSets</li>
</ul>
<p>Process</p>
<ul>
<li>input data is split into distinct sets - splits (one split is 16MB)</li>
<li>one process is the master (coordinator) - it assigns an idle processing not to do a work unit (either MAP on split or REDUCE on subset of keys of a map produced by split)</li>
<li>the output of the map stage is buffered in memory.</li>
<li>when the map output buffer is written to disk (flushed): the output is partitioned into disjoined sets by hash(key) % R (prepare for the next reduce action)</li>
<li>master then selects idle processor for reduce step - the input file has already prepared by previous stage</li>
<li>reduce step: it first sorts the value set of a single key value (in order to remove duplicate values)</li>
<li>reduce step each writes its own output file (why not a global output file?)</li>
</ul>
<p>Output: multiple files (one for each invocation of reduce step); - no need to combine into a global file: most often this is the input for another map/reduce program.</p>
<p>Data structures</p>
<ul>
<li>for each task (either map or reduce) keep the state (idle, in-progress, complete), for completed jobs keep the result file (intermediate file) and its size</li>
<li>master pings each worker machine, if it fails then running tasks (or completed tasks - the result is kept on local machine!) are reset to idle state and resubmitted.</li>
<li>handles dependencies (reduce task depend on data task) given that the intermediate file may be gone due to worker machine failure.</li>
<li>a single master process (still writes check points so that a failure will be apparent from the output file)</li>
<li>bandwidth is conserved by use of GPF (google file system) feature - storage of chunks locally on the same machine as the worker process.</li>
<li>
<p>backup tasks: some workers are very slow (bad disk causes long IO times/machine is loaded) so a few machines are very slow (few tasks at the end of a batch run take very long time to complete); therefore, at the end of a run they duplicate remaining tasks (often that results in significant speed up).</p>
</li>
<li>
<p>the result of each map subtask is subdivided into R reduce tasks, so we have to assign the intermediate result (key, values) to a particular reduce task. This is often done by applying hash on key value (hash(key) % R = indexofreduceset)
However, if key is URL then they may want to group all results of the same host, so they would need (hash(hostname(uri-key)) % R = indexofreduceset) as hash function (user may specify custom function)</p>
</li>
<li>
<p>combiner function: may have additional step; function for doing reduce on local machine (before sending it to reduce task, to reduce bandwidth (like combining away intermediate results)</p>
</li>
<li>reader: standard modes for interpreting input: key/value pairs, treat each line as input record, etc. cam specify additional optional component for customized parsing of input file.</li>
</ul>
<h2 id="google-file-system">Google file system</h2>
<p>http://static.googleusercontent.com/media/research.google.com/en//archive/gfs-sosp2003.pdf</p>
<h3 id="principles">Principles</h3>
<ul>
<li>done as user mode library (not in kernel); does not do POSIX file semantics (even can have read from stale regions, temporarily - no operations on directory tree - it is maintained implicitly … read on)</li>
<li>components fail frequent; resilience and monitoring are very important (commodity hardware) must do replication.</li>
<li>dataset: large files (not many small files); this means block size and IO size must be tuned to this (no optimizations for small files!)</li>
<li>mostly appending files; almost no overwrites of existing data.</li>
<li>most read operations are sequential (read 1MB or more) (because most programs in google are data analysis / passes on the whole data set); (some reads are from random offset - small in size : applications often batch and sort small reads - to avoid performance problems (minimize seek times?))</li>
<li>caching is not important - (that doesn’t help with sequential access).</li>
<li>how do application considerations influence priorities? - they added
append so that multiple clients can append simultaneously (for map/reduce, apparently), consistency model also designed per requirements of applications</li>
</ul>
<p>Services</p>
<ul>
<li>files are subdivided into equal sized chunks (64 mb) identified by 64bit handle. and multiple replicas of a chunks are maintained.</li>
<li>file: operations create/delete/open/close/read/write</li>
<li>snapshot create (fast copy of file or directory)/record append (allow to append concurrently - atomically)</li>
<li>file renaming atomically (as there is one master server that holds all metadata)</li>
</ul>
<h3 id="architecture">Architecture:</h3>
<p>Actors:</p>
<p>Master - holds metadata
Chunk server - holds chunk data (chunk is fixed sized range of 64MB)
Client - talks to Master to get lease info, then with that info does IO on chunk via Chunk server</p>
<h4 id="master">master</h4>
<ul>
<li>single instance</li>
<li>has metadata on all files hosted (file names, subdivision into chunks, where chunks are stored (set of chunk server addresses));</li>
<li>each file is divided into fixed sized chunks (64 MB size), each chunk is identified by a 64 bit chunk id</li>
<li>one master instance! no data is read through master (so that it does not become bottleneck, only metadata is handled);</li>
<li>all metadata in memory (no paging).</li>
<li>master periodically checks each chunk server for status (HEARTBEAT messages - the response includes which chunks are hosted by the chunks server instance (that makes relatively big messages!))</li>
<li>master handles replication of cached copies (of chunks among different chunk server instances).
(may start to re-replicate chunks upon to meet high loads of particular chunks)</li>
<li>master creates log of metadata changes (so that it can restore its state in event of master server crash)</li>
<li>the logs do NOT have info on location of chunks (on what machine a chunk is hosted); The master asks the chunk servers on start-up and gets this information through HEARTBEAT checks (can’t store this info persistently in master - a chunk server may get corrupted disk, or other permanent failures) chunk server join and leave the cluster anyway.</li>
<li>the metadata record log is replicated on several machines (global consistency!); user gets ack on request only when the log for that change has been stored persistently - in all replicas.</li>
<li>when log grows to large they put in a serialized dump of metadata (BTREE, this entry is called checkpoint) and start with the logs after that (deltas relative to checkpoint)</li>
<li>they do not block updates upon metadata flush (writing of checkpoint); if metadata changes need to be done while checkpoint is being created and written, then this change is written into separate log file instance for this purpose!
if incomplete checkpoint transaction then the tail is gathered from such a separate log file instance (but how is this temporal log file merged back into the main sequence when checkpoint has been completed?)</li>
<li>server cleans out stale regions (each region info returned in HEARBEAT has version tag - can have region that becomes invalid after update (update completed while a chunk server was down)</li>
<li>HEARTBEAT even has checksums on the data and tries to identify corrupted data so that it must replicate bad instances! (says so in 2.7.1) (? Limited guarantee - CRC checksums can go very bad with zero runs?)</li>
</ul>
<h4 id="chunk-server">chunk server</h4>
<ul>
<li>hosts chunk data</li>
<li>multiple computers that are communicate with master.</li>
<li>chunk server hosts several chunks, a client communicates with chunk server to read/modify data in a chunk.</li>
<li>responds to heartbeat messages sent by master to check chunk server status and send status of chunks)</li>
</ul>
<h4 id="client">Client</h4>
<ul>
<li>opens file with master;</li>
<li>obtains temporary leases on chunks from master (chunked = offset / CHUNK_SIZE); IO on chunk goes to chunk server;</li>
<li>no file data is cached by client, client caches metadata (what are files? where is the chunk?) for a limited period.</li>
<li>upon READ - translates offset to chunk id (= OFFET / FIXED_CHUNK_SIZE) - calls master for lease on chunk (and possibly several following chunks.</li>
<li>the server replies with location of all chunk replicas; the client chooses the nearest replica instance for read access.</li>
</ul>
<h3 id="consistency-model">Consistency model</h3>
<p>(now it gets complicated)</p>
<p>Consistent region := all clients see the same data on all replicas (failed mutation makes region inconsistent)</p>
<p>Defined region := (upon modification by client A) region must be consistent and all clients saw the modifications of client A (problem: concurrent modifications leave a region defined but not consistent - writes from different clients become intermingled)</p>
<p>? didn’t they just say that only one client can get a write lease at a time?</p>
<p>Types of modifications:</p>
<ul>
<li>Write - the application says what offset within a region will change</li>
<li>Append - an atomic ‘record’ append (? says at least once?) the offset of the write is determined by GPFS</li>
<li>Regular append - a write at the end of the file (at offset that is believed to be the end of file)</li>
</ul>
<p>A write has to be copied among all chunk servers to become ‘Consistent’ (? When is this replication done? upon each write or is there an explicit action like ‘release lease’ that triggers this action?)</p>
<p>Now each region instance has version tag; (this is also used to deleted ‘garbage’/’stale regions’ that are out of date (chunk server down while another party modified the chunk) (? So the tag must be if hierarchic nature to detect such situations? not just a version number?)</p>
<p>Problem: client may temporarily read from stale replica (this problem is not solved, but read leases are temporary so that the problem is not quite as bad, at least stale replica can’t get modified)</p>
<p>HEARTBEAT even has checksums on the data and tries to identify corrupted data so that it must replicate bad instances! (says so in 2.7.1) (? Limited guarantee - CRC checksums can go very bad with zero runs?)</p>
<p>Can have data loss if all replicas of a region are bad, but system identifies this situation.</p>
<h3 id="write-flow">write flow</h3>
<ul>
<li>for a file/chunk the client gets a write lease from server (granted to one client exclusively)</li>
<li>server lease response includes list of addresses of all chunk servers that holds the chunk</li>
<li>client writes the data to all chunk servers (that hold this chunk id); chunk servers buffers this data (no write yet - doesn’t have serial id)</li>
<li>when all acks are in: send write request to master server</li>
<li>master assigns serial id for this write; (master log records written - apparently) - master sends write request to all chunk servers (with serial number)</li>
<li>chunk servers respond with write status; the status is returned to the client; the write is considered failed by the client if at least some chunk servers had errors while writing the data to disk. In this event the write is repeated with the current lease (for a couple of times); If it failed for all retries then client retries the write op from from the beginning.</li>
</ul>
<p>Problems: write that cross chunk boundaries - are done as separate IO requests (each can fail, servers fail), so the file may be left in interesting global state (consistent but undefined)</p>
<h3 id="write-flow-for-record-append">write flow for record append</h3>
<p>Now record append may have a situation where some chunk servers failed, in this situation the client does some record padding and retries the write at a different offset within the chunk; (padding may be up to a third of chunk size! - if the record is near the end of the chunk). If write crosses chunk boundary (tricky cases reported by some chunk server instances) then part of the data must be rewritten in the next chunk (wow, application that uses this stuff must be really complex)</p>
<h3 id="communication">communication</h3>
<p>Actually the client is writing the chunk data to chunk server S1, S1 sends the data to chunk server S2 …. (each one chooses the closes neighbor (on the same link preferred, or smallest number of in-between switches). this saves bandwidth.
(data is pipelined, S1 forwards to S2 as the data is received!)</p>
<h3 id="snapshots">snapshots</h3>
<p>copying files just creates a reference (comparatively fast, still involves writing record to log), the actual chunk data is copied upon modification (copy on write)
Has to do the following:</p>
<ul>
<li>invalidates all outstanding leases (next interaction with the master will give it a chance to cope)</li>
<li>writes record to log</li>
</ul>
<h3 id="master-operations">Master operations</h3>
<p>many gory details here</p>
<h4 id="namespace-management">namespace management</h4>
<p>GFS has filenames like this aaa/bbb/ccc/ddd.file - it applies prefix compression, so that that common prefixes are represented as tree nodes, each tree node has a read/write lock.
Any operation (like obtaining a lease) needs to get a lock on every component of this implicit tree (therefore can’t create name component(aka directory) while a file of the same name is being written)</p>
<h4 id="where-are-new-replica-of-chunks-placed">where are new replica of chunks placed?</h4>
<p>policy must maximize both bandwidth utilization and reliability.</p>
<ul>
<li>Can’t place all replica on the same network link (rack) - if the switch is down then all replica are not available.</li>
<li>but some are on the same net link (this maximize bandwidth because data is routed preferably between instances on the same link)</li>
</ul>
<p>When creating new chunk:</p>
<ul>
<li>choose a machine with low disk utilization (to even things out)</li>
<li>limit number of recently created chunks on same machine (to prevent hot spots)</li>
</ul>
<p>Re-replication: copy existing replicates when some replicated data has become corrupted and we need to get target number of replicates.</p>
<ul>
<li>if a chunk has lost two replicas than it is in higher priority in the queue of pending replications than one that has lost one replica</li>
<li>chunks belonging to recently modified or active files is in higher priority for replication.</li>
<li>chunks ‘blocking client progress’ are in highest priority if replication needed.
Also rebalancing: moving chunks with aim of evening disk utilization/improving network utilization/</li>
</ul>
<h4 id="space-reclamation">space reclamation</h4>
<p>when file is deleted</p>
<ul>
<li>master writes metadata record</li>
<li>space is not reclaimed immediately! chunk server leaves the chunk data (may not be read via new leases - as the master has no metadata); chunk data is renamed to be a ‘hidden file’</li>
<li>space is freed by ‘garbage collection’: after 3 days the ‘hidden file’ get deleted and space can be reused.</li>
<li>during HEARBEAT: in each message the chunk server reports a subset of its chunks and the status/version; the MASTER reply contains the ids of chunk that are superseded by new version numbers, or where no metadata exists (orphaned). chunk server deletes these files</li>
</ul>
<h3 id="fault-tolerance--diagnostics">fault tolerance & diagnostics</h3>
<h4 id="fast-startup">fast startup</h4>
<p>both master & chunk servers start fast. no special abnormal termination scenario - all exits are equal.</p>
<h4 id="reliability">reliability</h4>
<p>chunk data is replicated, master log too.</p>
<p>there is a shadow executable of the master that only reads logs + is ready any time. If current master crashes, then the DNS server sets its record to the shadow copy of master. (new access to master via DNS lookup)
(? still it doesn’t know where the replica are - this info it gets when HEARBEAT messages arrives?)</p>
<p>On disk data corruption: chunk divided into 32k blocks, each block has its checksum (multi bit failures?)
chunk server checks this checksum against the checksum computed over data that is about to be returned.
write of new data is easy, overwrite of existing data needs to check the checksum of old data, overwrite then check checksum of newly written data (!)</p>
<h2 id="tao-facebook-distributed-data-store-for-social-graph">TAO: Facebook distributed data store for social graph</h2>
<p>https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf
https://www.usenix.org/conference/atc13/technical-sessions/presentation/bronson</p>
<ul>
<li>First they used SQL DB (mySQL) and Memcache as lookaside cache</li>
<li>walking the local environment of a graph is probably faster than ploughing through an SQL table in search of every relevant table entry
so they came up with a graph oriented database.</li>
<li>TAO still uses mySQL as persistent storage, but for the cache they now have a graph abstraction; (TAO is mostly read access, infrequent writes)</li>
</ul>
<h3 id="design-goals">Design goals:</h3>
<p>Optimized for reads / favor efficiency reliability over consistency</p>
<p>All objects are nodes in a graph; different object types - user node (each instance is a different user); checked-in-at (event type node); a comment is a node of type COMMENT.</p>
<h3 id="tao-concepts">TAO concepts:</h3>
<p>TAO objects are typed nodes in a graph;</p>
<p>TAO objects := each object has a unique 64 bit integer id, object type (otype)</p>
<p>Tao associations - typed direct edges in a graph (unidirectional link)</p>
<p>Tao associations := source object (id1), target object (id2), association type (atype)</p>
<p>type - determines what attributes
a object or associations can hold (there is a schema (type definition) that defines the set of attributes + default values)</p>
<p>All associations have a 32 bit time value (the schema for all associations has a time value)</p>
<p>Object: (id) -> (otype, (key value)?)</p>
<p>Assoc.: (id1, atype, id2) -> (time, (key value)?)</p>
<h2 id="tao-interface">Tao interface:</h2>
<ul>
<li>creation of new object/find & delete object by id</li>
<li>creation of link (also provides automatic creation/update/deletion of an inverse relationship instance - if inverse relationship is asked for/defined in schema)</li>
<li>no atomic compare and set (they do eventual consistency)</li>
<li>manipulating associations
<ul>
<li>assoc-add(id1, atype, id2, time, (k->v)*)</li>
<li>assoc-delete(id1, atype, id2)</li>
<li>assoc-change-type(id1, atype, id2, newtype)</li>
</ul>
</li>
</ul>
<h3 id="query-interface">Query interface</h3>
<p>Association list for object with id := list of edges (associations) that connect to object id, sorted by creation type</p>
<table>
<tbody>
<tr>
<td>Association List: (id1, atype) -> [ a - association (edge)</td>
<td>anew . . .aold]</td>
</tr>
</tbody>
</table>
<p>now they got queries to ask for the Association list given that you have an id (and other constraints: like link type and optionally a time filter.</p>
<p>The API only returns up to a maximum of objects - if there are more than this they need repeated queries.</p>
<ul>
<li>assoc get( /<em>in</em>/ id1, /<em>in</em>/ atype, /<em>out</em>/ id2set, /<em>out</em>/ high?, /<em>out</em>/ low?)</li>
<li>assoc count( /<em>in</em>/ id1, /<em>in</em>/ atype)</li>
</ul>
<h2 id="architecture-1">Architecture</h2>
<ul>
<li>use SQL DB (mySQLl) as backing store.</li>
<li>(important for such thing as backups / bulk updates)</li>
<li>use shards (DB is too big); each object id includes the ID of the shard that holds it (? can’t add any new shards?)</li>
<li>links don’t have a shard ID in their object id (they are tied to the shard of the object) links are stored on the shard of obj1 (source of edge)
(bidirectional links may need to update two shards (if id1 and id2 are in different shards) (No atomicity: if the backward edge was not written then this edge will be cleaned up by background task).</li>
</ul>
<h3 id="cache-layer">cache layer</h3>
<p>set of cache servers form a tier - a tier can handle any TAO request. Individual servers within a tear are sharded (similar to the DB sharding)
cache eviction - Least recently used entries get thrown out.</p>
<p>For each tear there is a cache leader server (cache server that does updates to from mySQL DB
There are second tear servers (followers) that are only in memory store, if cache miss then this one talks to leader server.</p>
<p>When leader fails: follows read directly from DB; for writes it sets a temporary leader (from among the followers)</p>
<p>when followers fail: For each tear there is a backup tier; When request to a shard server within a tear fail then request is still sent to corresponding shard server in backup tier.</p>
<p>Problems: consistency: an update in leader needs to send cache invalidation messages to each instance of following leader (change eventually will propagate - eventual consistency)
Each cached item has version, when leader gets modify request it sends update with version number (important: follower cache must check if its current data is not stale relative to this version number) The follower cache update must be completed before the write request returns.</p>
<p>Advantage: can add following leaders as much as required.</p>
<p>Each geographical region has its own replica of DB + leader + follower caches.
Now DB must be replicated across regions(!)
Leader server of each region can also talk directly to leader of each other region (!)
(says they have millisecond latency for communication between regions (!)</p>
<h3 id="caching-server---modified-memcache">Caching server - modified Memcache</h3>
<ul>
<li>manages LRU (least recently used) for objects of the same size, uses SLAB allocator (set of pools, each pool of fixed size allocations)</li>
<li>
<p>memory is divided into Arenas (some object+association types get into the same arena) - this is for ‘better isolation: allows to have different cache lifetime for important object types)</p>
</li>
<li>refrain from keeping small items in separate hash bucket; here they have a dynamic array per bucket.</li>
<li>active pairs of (id, atype) are represented by a 16 bit integer index.</li>
</ul>
<h3 id="mysql-schema">MySQL schema</h3>
<ul>
<li>all attributes of an object are in the same data column (do not have a separate table per object type)</li>
<li>for associations: must have range queries to that association table has an index based on (id,atype,time)</li>
<li>cache Sharding? (multiple follower caches for each Shard)</li>
</ul>
<hr />
Informal structures
2018-07-24T00:00:00+00:00
http://mosermichael.github.com//all/blog/2018/07/24/structure
["", "blog"] | <h2 id="informal-structures">Informal structures</h2>
<p class="publish_date">
24 July 2018
</p>
<p>A very interesting discussion of structure-less organizations ( <a href="https://news.ycombinator.com/item?id=17584563">here</a> ). Some software companies (noteable examples - valve and facebook - to some extent) try to break with the traditional forms of hierarchical organization structures and try to go with a more flat/self-organizing system. However this is quite tricky: the problem is that in the absence of formal hierarchies there will be informal ones.</p>
<p>The politics of weakly structured/structure-less organizations can be quite complex, Jo Freemain examines the experience of the feminist movement to this respect and explains the difficulties <a href="https://www.jofreeman.com/joreen/tyranny.htm">here</a></p>
<p>the main points</p>
<ul>
<li>people are social beings, therefore they are bound to organize, this implies some form of hierarchy, even if it remains informal.</li>
<li>if you have informal hierarchies then they have an interest to act ‘behind the scenes’ (so as to remain unaccountable!)</li>
<li>very astute observation on how to spot hidden networks of influence.</li>
<li>maintaining the informal links that make a weakly structured system go create a lot of overhead and lead to inefficiencies.</li>
<li>explains the rational of green movements org. structure (proposes measures that are designed to address the shortcomings, how to bring in structure without too much structure)</li>
</ul>
<p>(my notes on reading this text are given at the end).</p>
<p>Every type of organization can be managed either skillfully - that’s when the org. is succeeding at problem solving and generally growing, or it will have less capable manager that are not up the task.
I think that creating an organization that works well (any form of organization) is really a form of art - there seems to be some secret sauce not described in any books that makes it all tick. (for example even a dysfunctional system like this one somehow works for Valve, they seem to be doing fine from the business side of things)</p>
<p>The form or organization that is prevalent at any given moment: Probably the more practical system is selected as the prevalent model of organization - in any event there is some form of evolution that weeds out the losers. There is a lot to be gained by going with the prevalent system of organization- you can select for proven managerial talent for example;</p>
<p>however some experiments can succeed - you might be more likely to attract exceptional talent who might want to go for the lure of the untried, organisations like valve or facebook probably try to attract such people and try to put them into commanding positions - or rather hope that they will get a lot of informal power within the organization (don’t know if it worked out - never worked for them); however my guess as an outsider is that this system has its problems when it comes to growth and scaling.</p>
<p>The article by Jo Freeman recommends some measures that are supposed to avoid the shortcomings of structurelessness - however I think that these are measures designed to prevent an accumulation of power and information so that leaders remain interchangeable (like rotation); this does work against building individual competence. These are hard to maintain as a movement/organization gets bigger - for example the green party in Germany abandoned rotation and direct democracy at some stage.</p>
<p>From the position of an individual: given all this information, I think the crucial questions that one should try to figure out as an applicant when applying for a job at any kind of organization are as follows:</p>
<ul>
<li>how is work evaluated? Are there periodic evaluations of the results?</li>
<li>who is performing the evaluation?</li>
<li>how are conflicts resolved?</li>
<li>who is owning the code for a component? Who has the right to modify a given unit / component?</li>
<li>Processes: in what manner is there a planning of milestones? With what resolutions? Who is doing the planning?</li>
<li>to what extent are there informal groups that are governing the show? (ouch - you probably will not ask such a question at an interview, however there are means of checking that, for example if you have a series of interviews at any given company you can look up if there is a common background from the CVs of your interviewer - nowadays almost everyone is on linked-in)</li>
<li>processes oriented vs. goal oriented; don’t get me wrong - processes are very important, but the question is what counts in the end of the day? Can processes be adapted to fit a task, what amount of flexibility is there in the system? One way would be to figure out how the organization handled problems in the past - but I really would not know how to ask such questions in an interview.</li>
</ul>
<p>Of course who is going to hire you when you ask all these questions at an interview … (nervous giggle). Now go figure…</p>
<hr />
<p>notes om <a href="https://www.jofreeman.com/joreen/tyranny.htm"> THE TYRANNY of STRUCTURELESSNESS </a></p>
<p>by Jo Freeman aka Joreen</p>
<ul>
<li>structureless : the main form of organization in women movement.</li>
<li>was quite good in early stages: when the main goal of the movement was ‘raising awareness’ (it encouraged participation and discussion, resulted in ‘supportive atmosphere’)</li>
<li>
<p>problems start when groups went to do something specific: says the inherent limits of this form of organization showed up.</p>
</li>
<li>says some form of (informal) structure always shows up in any group of people that meet over time “it may evenly or unevenly distribute tasks, power and resources over the members of the group” - this because people are different “we are individuals, with different talents, predispositions, and backgrounds makes this inevitable” (likens it to futile pursuit of ideals like “objective” news story, “value-free” social science, or a “free” economy.</li>
<li>“Thus structurelessness becomes a way of masking power, and within the women’s movement is usually most strongly advocated by those who are the most powerful”</li>
<li>
<p>structureless as a form of deception: “As long as the structure of the group is informal, the rules of how decisions are made are known only to a few and awareness of power is limited to those who know the rules. Those who do not know the rules and are not chosen for initiation must remain in confusion, or suffer from paranoid delusions that something is happening of which they are not quite aware.” - says elites form out of the informal structure of structureless groups!</p>
</li>
<li>“an elite refers to a small group of people who have power over a larger group of which they are part, usually without direct responsibility to that larger group, and often without their knowledge or consent”</li>
<li>elitist advocate the rule of an elite</li>
<li>“The most insidious elites are usually run by people not known to the larger public at all. Intelligent elitists are usually smart enough not to allow themselves to become well known; when they become known, they are watched, and the mask over their power is no longer firmly lodged.”</li>
<li>“Elites are nothing more, and nothing less, than groups of friends who also happen to participate in the same political activity” “These friendship groups function as networks of communication outside any regular channels for such communication that may have been set up by a group.”</li>
<li>such a group may well form the elite of an otherwise unstructured group (similar to ‘old school’/’locker room’ mentality among other groups)</li>
<li>says if a group has two circles of competing friends this is “the healthiest situation, as the other members are in a position to arbitrate between the two competitors for power and thus to make demands on those to whom they give their temporary allegiance”</li>
<li>“At any small group meeting anyone with a sharp eye and an acute ear can tell who is influencing whom. The members of a friendship group will relate more to each other than to other people. They listen more attentively, and interrupt less; they repeat each other’s points and give in amiably; they tend to ignore or grapple with the “outs” whose approval is not necessary for making a decision.” … “Once one knows with whom it is important to check before a decision is made, and whose approval is the stamp of acceptance, one knows who is running things”</li>
</ul>
<p>Examines the history of women movement</p>
<ul>
<li>in early days of women’s movement married women were of higher standing (especially those married to new left man - they had command of resources that were important “and women were used to getting what they needed through men rather than independently”)</li>
<li>later they had a multitude of other different criteria for being in one of the elites (long list)
“The characteristics prerequisite for participating in the informal elites of the movement, and thus for exercising power, concern one’s background, personality, or allocation of time. They do not include one’s competence, dedication to feminism, talents, or potential contribution to the movement” - the background is determining your circle of friends! (the important thing here)</li>
<li>how to get ‘in’ : “The only main difference depends on whether one is in a group from the beginning, or joins it after it has begun. If involved from the beginning it is important to have as many of one’s personal friends as possible also join. If no one knows anyone else very well, then one must deliberately form friendships with a select number and establish the informal interaction patterns crucial to the creation of an informal structure. Once the informal patterns are formed they act to maintain themselves, and one of the most successful tactics of maintenance is to continuously recruit new people who “fit in.””</li>
<li>
<p>all this needs a lot of time; so says that formal structures are easier to join !</p>
</li>
<li>
<p>with ‘structureless’ groups all the power is with these informal in-groups who also lack accountability (because they claim not to exist at all) “informal structures have no obligation to be responsible to the group at large. Their power was not given to them; it cannot be taken away. Their influence is not based on what they do for the group; therefore they cannot be directly influenced by the group. This does not necessarily make informal structures irresponsible. Those who are concerned with maintaining their influence will usually try to be responsible. The group simply cannot compel such responsibility; it is dependent on the interests of the elite.”</p>
</li>
<li>women’s right movement does not elect/select speakers because of ‘structureless’ position: therefore ‘stars’ are called upon by the media to explain their position (still these ‘stars’ were never elected and are resented by the movement)</li>
<li>
<p>problems: ‘stars’ were not elected by the group and can’t be removed by the group!</p>
</li>
<li>
<p>Does it work? Most often not - but sometimes “the developed informal structure of the group coincides with an available need that the group can fill in such a way as to give the appearance that an Unstructured group “works.”“Still she identifies situation where it does ‘work’</p>
<ul>
<li>the task determines what needs to be done and when it needs to be done. It provides a guide by which people can judge their actions and make plans for future activity.
(the task is structuring the group)</li>
<li>group is homogeneous (is necessary to insure that participants have a “common language” for interaction) (diversity does not aid communication/understanding/common interpretation)</li>
<li>group must be small “Information must be passed on to everyone, opinions checked, work divided up, and participation assured in the relevant decisions. This is only possible if the group is small and people practically live together for the most crucial phases of the task. Needless to say, the number of interactions necessary to involve everybody increases geometrically with the number of participants. This inevitably limits group participants to about five, or excludes some from some of the decisions. Successful groups can be as large as 10 or 15, but only when they are in fact composed of several smaller subgroups which perform specific parts of the task, and whose members overlap with each other so that knowledge of what the different subgroups are doing can be passed around easily.”</li>
<li>low level of skill specialization - so that people become interchangeable (no one may be indispensable (why?)</li>
</ul>
</li>
<li>
<p>conflicts between multiple informal groups remain hidden (because the group do not want to go public - and are unable to acknowledge their own existence; the ideal is after all structurelessness)</p>
</li>
<li>
<p>principles for ‘doing it right’ (all copied from the text “</p>
<ul>
<li>Delegation of specific authority to specific individuals for specific tasks by democratic procedures. Letting people assume jobs or tasks only by default means they are not dependably done. If people are selected to do a task, preferably after expressing an interest or willingness to do it, they have made a commitment which cannot so easily be ignored.</li>
<li>Requiring all those to whom authority has been delegated to be responsible to those who selected them. This is how the group has control over people in positions of authority. Individuals may exercise power, but it is the group that has ultimate say over how the power is exercised.</li>
<li>Distribution of authority among as many people as is reasonably possible. This prevents monopoly of power and requires those in positions of authority to consult with many others in the process of exercising it. It also gives many people the opportunity to have responsibility for specific tasks and thereby to learn different skills.</li>
<li>Rotation of tasks among individuals. Responsibilities which are held too long by one person, formally or informally, come to be seen as that person’s “property” and are not easily relinquished or controlled by the group. Conversely, if tasks are rotated too frequently the individual does not have time to learn her job well and acquire the sense of satisfaction of doing a good job.</li>
<li>Allocation of tasks along rational criteria. Selecting someone for a position because they are liked by the group or giving them hard work because they are disliked serves neither the group nor the person in the long run. Ability, interest, and responsibility have got to be the major concerns in such selection. People should be given an opportunity to learn skills they do not have, but this is best done through some sort of “apprenticeship” program rather than the “sink or swim” method. Having a responsibility one can’t handle well is demoralizing. Conversely, being blacklisted from doing what one can do well does not encourage one to develop one’s skills. Women have been punished for being competent throughout most of human history; the movement does not need to repeat this process.</li>
<li>Diffusion of information to everyone as frequently as possible. Information is power. Access to information enhances one’s power. When an informal network spreads new ideas and information among themselves outside the group, they are already engaged in the process of forming an opinion – without the group participating. The more one knows about how things work and what is happening, the more politically effective one can be.</li>
<li>Equal access to resources needed by the group. This is not always perfectly possible, but should be striven for. A member who maintains a monopoly over a needed resource (like a printing press owned by a husband, or a darkroom) can unduly influence the use of that resource. Skills and information are also resources. Members’ skills can be equitably available only when members are willing to teach what they know to others.”
“</li>
</ul>
</li>
</ul>
Endianness
2018-06-03T00:00:00+00:00
http://mosermichael.github.com//all/blog/2018/06/03/endian
["", "blog"] | <h2 id="endianness">Endianness</h2>
<p class="publish_date">
28 May 2018
</p>
<p>keep getting mixed up on endianness - so here is a small test to show this stuff (to remind me not to check it again…)</p>
<blockquote>
<code><pre>
HOST BYTE ORDER - LITTLE ENDIAN
Number: 0x1020304
In memory:
00000000 04 03 02 01 |....|
00000004
NETWORK BYTE ORDER - BIG ENDIAN
Number: 0x1020304
In memory:
00000000 01 02 03 04 |....|
00000004
</pre></code>
</blockquote>
<p>Now host byte order (little endian) has the advantage that a uint16_t can be cast to uint32_t without changes;</p>
<p>Network byte order (big endian) is represented the same order as it is written (but the cast gets you the upper two bytes).
However with network byte order your specifications map directly to how it is represented (that’s why it’s called network byte order).</p>
<p>This endian naming thing just confuses the customer - host/network byte order is a natural description of what is going on.</p>
<blockquote>
<code><pre>
#include <stdio.h>
#include <stdint.h>
#include <netinet/in.h>
main()
{
uint32_t x;
FILE *fp;
x = 0x01020304;
fprintf(stderr,"HOST BYTE ORDER - LITTLE ENDIAN\n\tNumber: 0x%x\n", x);
fp = fopen("tr","w+");
if (fp != NULL)
{
fwrite(&x,sizeof(x),1,fp);
fclose(fp);
fprintf(stderr,"\nIn memory:\n\n");
system("hexdump -C tr");
}
fprintf(stderr,"\nNETWORK BYTE ORDER - BIG ENDIAN\n\tNumber: 0x%x\n", x);
x= htonl(x);
fp = fopen("tr" ,"w+");
if (fp != NULL)
{
fwrite(&x,sizeof(x),1,fp);
fclose(fp);
fprintf(stderr,"\nIn memory:\n\n");
system("hexdump -C tr");
}
return 0;
}
</pre></code>
</blockquote>
Things do change, invariably
2018-05-07T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2018/05/07/anotherana
["", "ramblings"] | <notextile>Things do change, invariably</notextile>
<hr />
<p class="publish_date">
7 May 2018
</p>
<p>Reviewing “Against the Grain - a deep history of the earliest states” by James C. Scott and “Ultrasociety: How 10000 Years of War Made Humans the greatest cooperators on Earth” by Peter Turchin</p>
<p>What is the nature of a state and where did it come from? Is it primarily a benevolent framework that
enables civilization or is it a mechanism of suppression? I guess you will get different answers here. Most Intellectuals are the heirs of the ancient scribes - for them the state is the source of order and personal income and therefore a blessing; however, there are those who do not concur, for Intellectuals are supposed to have their own independent opinion!</p>
<p>“Against the Grain” tries to explain the state by looking at it from the position of the outsider - the nomad.
Interactions between nomads and the state are quite complex (the author states that both the elites
within a state and nomads are benefiting from the surpluses produced by agriculture - one by
expropriation, the other by means of raids), however a look from a different direction is probably quite
refreshing to an age long discussion.</p>
<p>When Pinsker and Turchin speak of the early state, they speak of progress: it allowed for a reduced
level of extreme inter-personal violence (warfare) for an increasing population and gradually bigger
societies mean an evolving ‘superstructure’ and are just better! James Scott sees the opposite - the live
of a nomad or farmer under flood-retreat agriculture was much more varied and healthy than that of a
laborer in a Mesopotamian city; city walls are primarily designed to keep their own population within
the city wall (and to keep the enemy out of the gate); also, you need a great deal of violence to maintain
a regime where the majority of the population are slaves.</p>
<p>Peter Turchin says that the multitude of city-states in Mesopotamia led to an arms race between the polities - the
more populous cities with the bigger grain surplus were much stronger in this competition, so there was
a strong ‘selection pressure’ towards bigger polities with bigger grain storage facilities. This led to
more labor-intensive forms of agriculture and industry that given the could only be sustained by slavery,
given that both agriculture and industry were of low productivity.</p>
<p>Another guess is that flood-retreat agriculture had a serious problem: the floods brough in silt, and this was also leading to soil salinization (see <a href="http://faculty.bennington.edu/~kwoods/classes/enviro-hist/salt%20and%20silt%20in%20mesopotamia.pdf">Salt and Silt in Ancient Mesopotamian Agriculture</a> That would in turn require more labor intensive forms of irrigation, and these would drive more complex forms of centralization around temples (Mesopotamian temples were in charge of the irrigation effort, Kings with permanent palace structures were a later addition). Actually this possibility is no mentioned in ‘Against the Grain’.</p>
<p>Still, the collapse of a state often meant a return to smaller scale agriculture - that invariably would
mean the deaths of many people. Also there is this thing called slave mentality - a person in bondage
would be accustomed to his conditions; not for this fact only a few big slave uprisings are known
throughout history.</p>
<p>The transitions towards labor without bondage happened a few millennia later; when increased
productivity went in hand with more complex tools and social norms. A more complex environment
requires the presence of a motivated workforce, this motivation is impossible to create with slaves, who
are mostly passive. Again, the competition between states would favor those states that got rid of
bondage in favor of free labor.</p>
<p>Competition is a force of change: the nature of the change can be different - it can bring about a society
with more liberties or it can bring about more bondage; but things invariably do change.</p>
<p>Obligatory Jethro Tull song: heavy horses</p>
<iframe width="420" height="345" src="https://www.youtube.com/embed/_GL6F2M7-gA" frameborder="0" allowfullscreen=""></iframe>
<pre>
“Heavy Horses, move the land under me
Behind the plough gliding - slipping and sliding free
Now you’re down to the few
And there’s no work to do
The tractor’s on its way”
</pre>
<p>My Questions while reading this text:</p>
<ul>
<li>was the state a generally less violent place than the non-state fringes? The author denies this (in a
different talk). Go figure.</li>
<li>what was the real level of violence in early states? (it can be that it was much more violent - as it had
to take measures to keep its population in - just like communist countries did)</li>
<li>nomads were working less than peasants in the service of the state? Makes sense - the peasant has to
maintain the whole edifice of the state by means of his surpluses; However, he does not get protection
from law/law enforcement; needs to stand his own ground and be like a criminal - maintain his own
law and order. So again the claim that level of violence is not too different does not hold up.</li>
<li>maybe the advances of the state also led to a less violent nomad society: More diverse and reliable
income sources would mean that there are less reasons for fighting over resources - that means more
loot (from looting) was make a more relaxed lifestyle. (Also nomads in a resource rich delta would
have less to fight over than people in a savanna; so the level of violence might be in directly inverse to
the risks of sustenance/availability of resources)</li>
<li>maybe its a general law: improvements of productivity lead to a more humane society by creating less
reasons for fighting over resources (examples: abolition of slavery)l (Now the great crisis of 1928 that
lead to WWII was a financial crisis, political crisis or crisis of overproduction ? Go figure. For every
rule there is a counterexample).</li>
<li>are the criminal classes the new nomads? (they are not part of established hierarchies, form structures
that are parallel to the state, also can have similar ‘protection’ roles as nomads did). However they are
much more violent than a state. (is it possible to say that early nomads were also much more violent
than nomads that live next to a modern state?)</li>
<li>how could a decentralized polity deal with the ‘tragedy of the commons’ ? (depletion due to
overcrowding)</li>
<li>If the Sumerian state came only that late, then how to explain the millenium of earlier proto writing
systems? (these were first done with the purpose of record keeping - so the state must have emerged a
bit earlier)</li>
<li>Nomads as the dark twin of the state (vying to appropriate the surplus of the grain core): I think they
could compete with a state until the industrial revolution - then the norms of production became so
demanding that the nomads couldn’t keep up on equal terms) A tribe just could not stand against a
disciplined army any longer (and discipline came from production regimentation) - I think that
firearms alone can’t explain this switch.</li>
</ul>
<hr />
<p>Notes on “Against the Grain - a deep history of the earliest states by James C. Scott”</p>
<ul>
<li>
<p>historical discourse is dominated by state records and narrative of progress; is this really the whole
picture?
(Progress is read as progression from hunter/gatherer to agriculture/state formation (and as a result of
state formation). Is that the case?)</p>
</li>
<li>Many possible kinds of agriculture - sedentism (staying on the same spot, required for state building) is
not required
<ul>
<li>sawing and moving around during harvest time</li>
<li>using wild wheat (not domesticated) possible too (what now counts as signs as domestication (larger grains, rachis that is not brittle) has been acquired later; there was no one domestication event: it was scattered over long time periods and different between regions.</li>
</ul>
</li>
<li>“Early Near Eastern villages domesticated plants and animals. Uruk urban institutions, in turn,
domesticated humans” (in terms of ruling / controlling / breeding)</li>
<li>early states were territorially quite small (as compared to area inhabited by tribes); population of
tribes was larger than that of the early states (that’s why they were such a potent military threat);</li>
<li>state was brittle at first: consolidated (centralized) + effective rule was not the norm (example: Greek
dark age); frequent challenges in the form of war and diseases. Also threat of raids (barbarians at the
gate): the city state had to either invest in defense or pay tribute (very costly)</li>
<li>However relationship of state and barbarians was symbiotic: trade (state got a lot of materials,
barbarians got manufactured goodies); Usually the state was shadowed by the rise of a barbarian twin
(also they sold their own as slaves and were mercenaries - the barbarians contributed mightily to the
decline of their brief golden age.)</li>
<li>many aspects of the life in states was undocumented (diseases, extent of slavery/bondage, number of
escapes from city towards wilderness)</li>
<li>(all very different from the triumphalist vision of progresses inherent in the state form)</li>
<li>
<p>the state form became dominant only around 1600! (when the nomad could no longer effectively
threaten it?)</p>
</li>
<li>the experience of a forager/nomad is much richer and healthier than that of full time agriculture in an
early state (state was built around grain, mobile nomad had richer and more various food sources and
life experience) ; (guesses that coercion is the basis, maybe because population was hemmed in by
surrounding deserts or other hostile periphery)</li>
<li>grain hypothesis: all early states arose around grain because its suited for tax collection (can’t do that
with tubers) needs determined harvest (tax collection time); easy storage: only grain has that.</li>
<li>the early state needs to get hold of land, get involved in landscaping (canal repair, silting) so that
population and area is ready/standardized for uniform grain production. (often fails due to problems)</li>
<li>defines state as a social stratification with king at the top, walled cities, tax collectors (UR III) . Says
that fixed settlements like Jericho do not fit this description.</li>
<li>a large bonded population was needed for canal construction/maintenance & agriculture; lots of
coercion to keep them in (the wall of china was built to keep enemies out and tax payers in)
examples helots in Sparta, frequent population flight in Babylon… violence was part of the system.</li>
<li>some reasons for state collapse
extrinsic (drought/climate change)
intrinsic</li>
<li>diseases due to lifestsyle (crowding/bad nutrition)</li>
<li>deforestation and resulting siltation/erosion</li>
<li>salinization of soil and resulting lower yields</li>
<li>collapse? If a kingdom was a loose confederation then that means temporary dispersal and later
possible reunification. If barbarians conquered the lands then they might also adopt the culture of the
rulers (see Babylon).</li>
</ul>
<p>Chapter 1</p>
<p>fire appears 400.000 years ago; now the Human becomes master of the cave! first tool for shaping the
landscape (burn it off) Native Americans burn wood areas off so as to attract game that they hunt. Also
used as a tool for hunting (drive animals off the cliff) - before bows and arrows!
(Even say that the little ice age (1500-1800) was because forests grew as Indians died of the plagues
brought in by Europeans so they did not burn down the trees as they used to do).
(other tools of landscape molding - selective hunting; moving eggs of birds, rearing young animals
etc.).
cooking had a big effect: we get concentrated food. can get more food sources (it makes it worth to dig
up;
brains became bigger as a result of cooking! (need less calories for chewing and needs a gut three times
smaller than chimpanzee)
In turn we are dependent on fire - so that it ‘domesticated us’</p>
<p>Timeline:</p>
<ul>
<li>before 10.800BC - trend of population growth and settlements</li>
<li>10800-9600BC - cooling (break - period called ‘younger Dryas (estimate of world population - 4
million)’</li>
<li>9600BC - gets warmer, and better (evidence of yearlong settlement)</li>
<li>6000-8000BC domestication of ‘founder’ crops and animals, first ‘urban agglomerations’ (what a word)
<ul>
<li>‘proto urban settlements’ around 6500BC (Alluvium) - were around wetlands</li>
<li>(a cold century in 6200BC)</li>
</ul>
</li>
<li>Note that these early settlements predate grain and livestock (!) (contradicts the assumption that grain was a prerequisite for settlement - and also they did not need irrigation projects as drivers - they were in wetlands - like Jericho at the time, and others all other early settlements as well….) during the Ubaid period (6500-3800BC) had a wide source of food (not just serials!) Says that alone hinders the establishment of centralized political control. Also had transportation by boat and trade (!)</li>
<li>5000BC estimate of world population - 5 million</li>
<li>3200BC - Uruk has population between 25000 - 5000)</li>
<li>
<p>0BC estimate of world population - 100 million)</p>
</li>
<li>
<p>Still states must drain marshes and irrigate arid areas (areas can’t be governed, based on commons of
the wetland, no hierarchy (no differences in burrial))- to grow wheat (that’s taxable)</p>
</li>
<li>
<p>Took 4000 years before we got to an early state - highly stable state of affairs! Agriculture was not the
main source of calories. Lots of subsistence strategies and food sources - as the weather and rainfall is
very variable.</p>
</li>
<li>People had shifting production roles and food sources; stigmatization of pastoralia came later - a
comparative reading of Gilgamesh: early versions Enkidu is just a pastoralist; 1000 years later he is a
subhuman barbarian (does not know to ‘bend the knee’) that has to be domesticated</li>
</ul>
<p>How did we get started with agriculture?</p>
<ul>
<li>Traditional view: grains can be stored over long time, so they are an ‘insurance policy’ for bad times</li>
<li>others say overhunting + population pressure lead to ‘broad spectrum revolution’ : a first wave where
agriculture became the dominant food source (took several thousand years) (evidence of nutritional
changes in bones) (wide spread anemia - grain based diets did not have iron or omega 3).</li>
<li>
<p>says the reasons are not clear: more plants after dryas?</p>
</li>
<li>social evolution says that agriculture is a planned activity with ‘delayed gratification’ (as opposed to
hunting). This ignores landscape sculpting and big hunts - these need planning too!</li>
<li>‘flood retreat’ agriculture demands few labor (put seeds in silts put up by flood). Says that’s the earliest
form suitable for hunter/gatherers.</li>
<li>still by 5000BC we had lots of villages that were farming cultivated wheat (labor intensive) as their
main food source. Why?</li>
<li>‘back to the wall hypothesis’ combination of population growth, less wildlife, a bit of coercion did the
trick. (like Adam and Eve being expelled from the garden of Eden to toil the land)</li>
<li>says evidence contradicts this theory: agriculture came first in areas of abundance. Less labor-
intensive Flood-retreat agriculture may have been the start.</li>
<li>how were grains domesticated? Natural selection works against desired traits - (favors small grains
and grain heads, easy to shatter grains)</li>
<li>says early grains were selected for resistance to parasites and stresses, to a lesser extent for yield (!)
only later came the fully domesticated strain that is fully dependent on our attention (and more labor
intensive).</li>
</ul>
<p>Domestication of animals.</p>
<ul>
<li>says species that had preadapted where domesticated: needs herd behavior, social hierarchy, muted
fright and flight response, can tolerate crowding, wide range of food, etc. (however gazelles, who live
in herds can’t be domesticated),</li>
<li>new kinds of selective pressure: people cull males and animals past reproductive age (bone remains
are judged by age to see if they come from wild or cultivated flocks)</li>
<li>domestication took short time - experiments with taming foxes says it takes 20 generations to get 35
tame foxes + physiological changes and lower adrenalin.</li>
<li>breeding gets less aggressive + more tolerant of other species; also get less male/female differences,
more fertile, decreased brain sizes (for tameness) !</li>
<li>brain areas affected - limbic system (responsible for hormone and reactions to threats/other stimuli)</li>
<li>mortality of offspring is much higher than with wild species (crowding, feces) but is offset by higher
fertility and younger reproductive age.</li>
<li>also bones of both man and sheep in cities got smaller with signs of nutritional distress (iron
deficiencies)</li>
<li>speculates: can it be that emotional live of man declined? (just like the reaction of animals) - Can’t be
answered objectively …. (only 240 generations have passed since agriculture started, so its hard to
generalize)</li>
<li>says man has been domesticated by agriculture (as co-dependents of the plants that we grow - with all
their special requirements)</li>
<li>hunter gatherers have lots of varied activities - all attuned to the rhythms of nature (animal migration,
gathering of wild fruits - all dependent on the time and cycles of the year). Each activity needs its “tool
kit” and techniques, each activity needs planning/coordination with other group members, also it needs
a lot of detailed knowledge (naturalists have taken note of that!).</li>
<li>
<p>in contrast agricultural life is attuned to just the clock of the major cereal (less complex by a margin)
dominated by public ritual “a time to sow and a time to reap”.</p>
</li>
<li>
<p>period from 10000BC - 5000BC population did not grow, in spite of invention of agriculture.
Speculates that his was because of diseases: ‘the most lethal period in human history’ due to infectious
diseases (and people were not used to them!). Problem: infectious diseases leave no marks on the bones,
so it is hard to prove.
Evidence to support this: frequent abandonment of settlements that does not correlate to cooling of
weather. (mention of diseases in Gilgagmesh epos, lots of amulets and ceremonies against diseases - it
must have been significant). In Mesopotamia they understood contagion and quarantined entire
quarters of a city!) and long-distance traders were kept out of the city - as potential carriers of disease!
(Akkadian word for disease means certain death)
Many disease vectors due to crowding, gathering of feces, mosquitoes (all multiplied by immobility).
(American natives crossed the Bering strait in 13000BC - before the plagues did develop due to
crowding. That’s why they all died from imported plagues - imported from Europe)</p>
</li>
<li>parasites/fungi developed as a threat to the grain monocultures. Argues that crop failures were
frequent and not understood at the time.</li>
<li>crops have to be defended against weeds, snails/rodents/birds (that’s why its so labor intensive); still
they developed techniques like crop rotation and scattering of fields (so as to protect against contagion)</li>
<li>
<p>It’s a small wonder how agriculture became the dominant mode (has the advantage that it can feed
greater number, but its so much more fragile…)</p>
</li>
<li>
<p>reproductions: farmers have a higher rate of reproduction (kids can be weaned earlier, it’s better to
have more working hands, fertility is higher / women remain fertile for longer); with nomads they can
have only few kids due to mobility requirements… this leads to a small percentage wise advantage for
agriculture - and that accumulates over time.</p>
</li>
<li>agriculture already existed when the first states appeared, so it was not a result of state building.
Rather says the state took advantage of the ‘grain and manpower module’ as the basis of its power grab.</li>
<li>how does he define ‘the state’ ?
<ul>
<li>needs officials that collect taxes and are answerable to ‘the ruler’</li>
<li>exercise power in hierarchic societies (with a division of labor)</li>
<li>needs an army, temple, palace + king, city with a wall (that’s Babylonia in 2800-3200BC, Uruk was
first with the wall business, population tripled over 300 years - must have brought in slaves/prisoners)</li>
<li>how comes? 2500-3200BC had declining levels of sea level + Euphrates, increasing aridity +
salinization; irrigation became labor intensive (needs canals)</li>
<li>says rich producted (flood deposited) alluvial or (wind deposited) loess soil are strictly required to
feed the superstructure of a state. Needs lots of imported goods/materials to function (wood, metals) -
so it also needs water transport! (Jerusalem is not near a river!)</li>
<li>first states were not formed in river delta’s, they were the site of the first cities but not of the first
states (Egypt, Mesopotamia, China) - although these areas are abundant in food. (? the alluvial soil
area is somewhere else ?)</li>
<li>serials (wheat, maize, rice) are the basis of the state because they ripen at the same time (yield can’t be
hidden from the tax collector) and are taxable, and easy to store) (Tubers like potatoes doe not have to
be gathered at the same time, hard to extract and to transport, hard to assess the yield - its underground)</li>
<li>states didn’t like/trusted merchants - they can hide their wealth.</li>
<li>serial crop farmer can evade taxes by harvesting not fully ripe grains, and to plant before the official
time (the state always does make an effort to standardize planting time / calendars are really important!
(easy for rice as it depends on flooding))</li>
</ul>
</li>
</ul>
<p>Walls make states</p>
<ul>
<li>Mesopotamian cities were walled. In epic of Gilgamesh walls were erected to protect the people.
When a city was defeated its walls were destroyed - no more agriculture around the place.</li>
<li>
<p>but says these walls also had to keep the population from running away! A city wall defines the limits
of political control. (One of the hallmarks of early statecraft in agrarian kingdoms was to hold the
population in place and prevent any unauthorized movement. Physical mobility and dispersal are the
bane of the tax man”)</p>
</li>
<li>states are record keeping machines (as a means of controlling the populace); peasant revolts always
burn down the office where official records are kept.</li>
<li>proto-writing systems first appeared as a way of keeping records (instrument of state building); 500
years before writing texts. Symbols in Sumer were the rod and the line (tools of the surveyor).</li>
<li>record Uruk IV (3100-3300BC) was all lists of grain, slaves, labor (they had work points - to
designate units of works, two liter sized bowels for grain (to measure food rations))</li>
<li>says URIII was a heavily centralized economy</li>
<li>Similar in China (though first signs of writing were used for divination). Qin: had a land registry that
recorded owner, rainfalls, quality of soil (so as to compute expected yield for taxation) (see equal field
system https://en.wikipedia.org/wiki/Equal-field_system )</li>
<li>super centralized early states did not last long (Ur III - 100 years, Qin - 50 years)</li>
<li>as the state disappeared so did literacy (was limited to officials). Fringes did not accept literacy out of
fear of control/subjugation that comes with it (the taxman)</li>
</ul>
<p>The Question Of Surpluses</p>
<ul>
<li>subjects need to be forced to produce surpluses (once self-sufficiency is achieved people do not
produce more on their own - nobody likes to work too hard (Chayanov)).</li>
<li>bondage/force was required to keep people from escaping to the woods. (that was relaxed when land
ownership was a universal concept - nowhere to run to).</li>
<li>state had to be careful: not press it too hard so that people would not have a strong desire to get out.
Were at a constant fight to fill losses due to people running away. (Babylonian legal code is full with
what to do with runaways)</li>
</ul>
<p>Slaves</p>
<ul>
<li>
<p>population was a reliable measure for a states wealth/military strength. Wars were more about
acquiring population (for transfer into the winner city) than about acquiring territory. (most wars in
Mesopotamia were about grabbing a smaller community rather than full wars between cities)</p>
</li>
<li>up to 1800 a third of the worlds population was in bondage (slavery or serfdom)</li>
<li>no slavery, no state; In Athens it is estimated that two thirds of all people were slaves.</li>
<li>what about slavery in Mesopotamia? The consensus is that they were fewer in numbers/of minor
importance (author thinks otherwise) - says its important because slaves were producing textiles (the
main export good) (in Uruk - 9000 worker, some debt slaves, some POWs); used for canal digging
and wall building. In Uruk they got ‘houses of prisoners’ with slaves - these were hired out in small
groups (so that they would not escape)) (Still here was a smaller total number of slave than in Greece
or Rome)</li>
<li>seems that Uruk scribes counted slaves as similar in status to domesticated animals, frequent mention
on cylinder seals of troops that are clubbing prisoners; reports of blinding prisoners (not clear how
common). Work houses were slaves were rented out in small number (so as to avoid revolts).</li>
<li>lots of effort to keep them in (bounty hunters of escaped slaves in URIII; Hammurabi code mentions
punishment for escaped slaves; depiction of slaves in neck fetters; also slaves did not reproduce in
captivity (probably died from overwork) (? much later: Hebrew slaves did manage to return from
Babylonian captivity?)</li>
<li>not clear if they had slaves in the old kingdom of Egypt; seem to have chattel slavery during the
Middle kingdom (demand for shackles for POWs) (first recorded strike during Ramses III - after
rations were not delivered to slaves, and scribes!)</li>
<li>
<p>Qin and Han dynasties: says cultivation was done mostly by slaves and ‘criminals’; slave markets;
Also resettled population of conquered areas - an even put up replicates of their temples!</p>
</li>
<li>states made an effort to capture skilled workers during conflicts (slavery as HR strategy).</li>
<li>records of wars often mention number of prisoners taken (and their skills!) - don’t mention the
captured territory in detail - the bounty is what counts.</li>
<li>these workers are detached from their society of origin, so they can’t form an effective opposition
(Janissaries, court Jews)</li>
<li>In Greece and Rome the huge concentration of slaves led to revolts (Babylon and Egypt did not have
this concentration of slaves)</li>
<li>women slaves; in Babylon and Egypt the children of slaves were free (unlike Greece, Rome). Women
slaves were valued (because of child mortality they needed lots of offsprings)</li>
<li>other uses of male slaves - mining, quarrying, timber (gulags with high mortality - far from the
household - not a political danger?)</li>
<li>slaves were an important supplement to the demographically challenged state - (“to extract such labor
exclusively from their own core subjects, they would have run a high risk of provoking flight or
rebellion”)</li>
</ul>
<p>Mesopotamian states had smaller number of slaves because</p>
<ul>
<li>did not have the geographical reach of Athens or Rome; therefore</li>
<li>neighboring polities were quite similar in culture, so captives would assimilate in a couple of
generations (women slaves and children)</li>
<li>special categories: Metics, ex slaves - 10 of population with restricted rights; Did Babylonia have a
similar concept?</li>
<li>says a substantial population of slaves was far away from urban centers (Mesopotamia had less of
than compared to Greece) - working in quarries/mines/forests.</li>
<li>says forced resettlement (Assyrian empire was huge operator here, probably moved to fill abandoned
areas<img src="" alt="" /> were probably known earlier and were practiced</li>
<li>‘the central role of bondage and coercion in creating and maintaining the grain-and-manpower nexus of
the early agrarian state would be perfectly evident’</li>
</ul>
<p>stability and collapse - what are the reasons for collapse?</p>
<ul>
<li>early pre-state settlements were frequently abandoned - very fragile.</li>
<li>early states were very fragile entities too) (collapse of UrIII (they lasted for 100 years, that’s a lot!)
Old Kingdom collapsed, etc. etc.) Comes with abandonment & dispersal of population centers (+ no
long-distance trading, no records kept, elites disappear; means decentralization and less hierarchy (not
necessarily collapse of population numbers (? But not after Rome?) (that’s what they mean by ‘collapse’)</li>
<li>archeology focuses on the state and its records; now we know to look at the larger picture: from
1800BC-800BC the big cities in Mesopotamia got to 1/14th of previous size (and fewer of them -
1/16th of previous number!) Don’t know why.</li>
<li>speculates that long distance trade in ‘Uruk world system’ brought in diseases that lead to the downfall,
warfare with population movement also move disease vectors (like the Romans were did in by plagues,
and the Mongol realm with its silk road).</li>
<li>other factors: soil salinization, erosion and deforestation (wood needs to be transportable, deletion
within that range leads to scarcity) The more charcoal is found in a layer - the more deforestation
occurred</li>
<li>land clearance/deforestation leads to malaria</li>
<li>diminishing yields due to problems may lead to ‘fading away’ of the state</li>
<li>also war between states , or it pays tribute)</li>
<li>waging war contributes to fragility of the state ; also a
defensible position for the city is chosen</li>
<li>states that loose wars will have manpower running away</li>
<li>size of the state depends on transportation costs ;
whoever the reach can be greater with water transport ! This leads to concentration of
resources around palace/temple complex and its vulnerability</li>
<li>early states lacked detailed accounting of yields</li>
<li>in times of crisis over-taxation then becomes necessity~~ leading to further manpower flight (leading
to collapse of the grain core)</li>
<li>crisis often coincides with demands to increase production (shorten fallows and demand for more to
be planted)</li>
<li>hard to estimate dynamics (population flight vs increase due to captives, etc.).</li>
</ul>
<p>Praising collapse</p>
<ul>
<li>says the period without a state (intermediate period, dark age) wasn’t that bad; better for most people (?
what about the recorded lamentations to that respect?) - high state period (Akkad, UrIII, Babylon - was
short compared to ‘intermediate periods’ (!)) (shortest interlude in Egypt - 100 years of intermediate
period between old and middle kingdom)</li>
<li>says a more dispersed population was less prone to diseases or wars, so overall a lot of lives would be
spared (? how is the same limited ecology supposed to support the same number of people living a
dispersed lifestyle?)</li>
<li>even culture survives the dispersal to multiple centers.</li>
<li>example: Amorite rule after Ur III: new rulers reduced taxes and forced labor - but encourage free
farmers and merchants. People dispersed from cities, but did not vanish (?) ; Intermediate period in
Egypt did not come with sharp population decline (?) (still not much is known as there are no records
(catch 22BC)) but with decentralization (regional centers took over instead of centralized realm)</li>
<li>‘golden age of barbarians’ state was a great target for raids - when the state was not able to impose its
will on barbarians; also trade with states was very lucrative for ‘barbarians’ (he calls them non state
people - their position is primarily defined in relation to the state, not in ‘culture’ (as they can be
incorporated to varying degree by a state) ‘They were, by virtue of their mobility, their diverse
livelihoods, and their dispersal, unsuitable raw material for appropriation and state building, and it was
for precisely these reasons that they were called barbarians.’ (?but the late Romans still made a
distinction based on origin !)) Says that’s wrong because people did switch between state and nomad
live stiles (state foundation and dispersal in times of crises, or flight from arbitrariness of state) (lots of
Romans would find Huns and Gothic rule preferable to Roman taxes and arbitrariness of elites - Totilla
promised freedom and land ownership - and got many Roman conscripts that way)</li>
<li>raiding had its problem because of ‘killing the goose that lays the eggs’. Instead they also did
protection rackets (require stable military/political environment) (form of gifts in return for formal
submission or payment for protection from other ‘tribes’ or payment for trade monopoly) Schemes like
this were frequent - and were often kept secret (to protect the standing of the state) Says both
‘barbarians’ and the state would exploit the grain core (base of accumulation) (state and its barbarian
twin were both as protection rackets to the agrarian core - taking slaves etc. (Mongol expression for
agrarian population: ra’aya, “herds.”)</li>
<li>in late antiquity there was a large volume in long distance trade (grain from Egypt); this ability to
transport goods made stuff collected at fringes more important (foraging boom) (bee wax, lumber,
aromatic woods, etc.), thus further integrating the non-state fringe.</li>
<li>when the state is in problems then it presses harder for taxes and conscripts - that makes people run
away (downward spiral) (but early palace states would have walls for keeping people from running
away - so that didn’t work either?)</li>
<li>geographically the early state was limited (to soil suitable for intensive agriculture) (that is up to the
middle Roman period where ships could transport large quantities of grain over a distance); all other
areas were ‘barbarian’ by definition.</li>
<li>non-state entities were the dark twins of the real state; complex relationship - sometime a state would
conquer and extend its rule (Rome and Gaul); sometimes tributes were extracted from the state;
sometimes a state was conquered (but then the invaders turned rulers); sometimes nomads became the
cavalry of the state.</li>
<li>Why did the ‘golden age of barbarians’ end around 1600? ‘As states and durable gunpowder empires grew, the ability of nonstate peoples to raid and dominate small states shrank at a pace that depended greatly on the region and its geography’. Still ‘barbarians’ were all slavers - and this trade made the core state stronger (at the expense of the fringes); they were often paid mercenaries of the state - this further eroded their own independence from the state.</li>
</ul>
<hr />
<p>Other authors do have a totally different perspective and interpretation;</p>
<p>See Peter Turchin’s “Ultrasociety: How 10000 Years of War Made Humans the greatest cooperators on Earth”</p>
<p>The author maintains that early societies were very violent places - graves from this period have a lot of signs of inter personal violence. The early city-state reduced this level of violence and channeled it into conflicts between states - these are fought by a small number of warriors (well, all-out conflicts like WW2 on the eastern front don’t quite fit in with this picture).
The polities were competing with each other and had to become better at mobilizing the cooperation of the populace and had to get better at organizing this cooperating in general.</p>
<p>here are my notes from reading this highly interesting book:</p>
<p>==============</p>
<p>notes on “Ultrasociety: How 10000 Years of War Made Humans the Greatest Cooperators on Earth” by Peter Turchin</p>
<p>‘cooperation is actually astonishingly difficult to achieve and, once achieved, hard to preserve’ - is brittle, can always fail.</p>
<p>For most of humanities history we had small tribes – everyone knew each other; how did human society change to big hives?
lets measure cooperation by looking at big projects:</p>
<ul>
<li>ISS - 3 million people years - a billion tax payers funded that.</li>
<li>Gothic cathedrals – 15.000 people years (300 builders working over 50 years) - several hundred thousand supporters/tax payers.</li>
<li>Colosseum - 12 thousand slaves for eight years - 30 million sesteros (they got that by looting/sacking Jerusalem)</li>
<li>Great pyramid of Giza – 400.000 people years.</li>
<li>Goebleki Teppe - neolithic structure - ??? (no written records)</li>
</ul>
<p>Goebleki Teppe - (estimated 240-300 people years) purpose was to bring people together (and build community and cooperation) - and then have festival with drink beer together.
How did they build it? (small number of people over long period or lots of them over short period?)
Poverty Point in Louisiana (1000 people years) - another neolithic structure was erected over short time (otherwise the rain would have washed away layered levels of different kinds of soil)</p>
<p>Classifies polities by their size</p>
<ul>
<li>foraging bands – 10s individuals</li>
<li>farming villages – 100s</li>
<li>simple chiefdom (several villages under one chief) - 1000s</li>
<li>complex chiefdom (three subordinate chiefs) - 10000s.</li>
<li>archaic states – 10000</li>
<li>macro-states – 1.000.000</li>
<li>mega-empires – 10.000.000</li>
<li>large-nation states = 100.000.000</li>
</ul>
<p>Between 1500BC and 1500BC they had mil.tech. based on adoption of horses – open plains with nomads would favor intensive warfare, hilly terrain would inhibit that.
They modeled spread of agriculture and growth of states – and warfare would be a decisive part of the model.
War as ‘destructive creation’ - it prompted institutes that support cooperation. (In Hinduism Shiva is both god of war and creation)</p>
<ul>
<li>tribal societies were very violent (significant number of remains have marks of violent death); the ‘peaceful savage’ is a myth.</li>
<li>(still tribal society is very cooperative – easy to organize collective action in small groups; the wonder is that large groups are still cooperative – yet some have to sacrifice for this (like cops))</li>
<li>as society became more complex violence became more ‘controlled’ (and there is less of it). what happened?
Many social theories: agriculture, hydraulic theory, conquest as social change, Marx: elites expropriate surplussed.
Then biology came in: (attempts to unify by evolution) Darwin said that morality gives a huge advance (‘The Descent of man’); also there is social Darwinism;
Wilson: behavior is regulated by genes (Sociobiology); cultural evolution (they like math models here)</li>
<li>instead of asking ‘how did empire X fail’ he is asking ‘how was empire X possible?’ - at each growth step arrangements had to be found to allow coop. with strangers.</li>
<li>after empire gets big internal cohesion declines (not needed as wars have been won); elites no longer restrained by common goals grab it all, cooperation declines; empire declines.</li>
</ul>
<p>Enron: says CEO Skilling instituted internal in-group competition (lowest performers got fired – rank and yank) that destroyed cooperation and bred cheating/unethical behavior towards customers.
Says ‘rank and yank’ is common practice – began in 70ies and 80ies when ‘unwritten compact previously existing during a past period of growth and progress’ was ended; Putnam: civic engagement lasted 30 years after New deal.
“features of social life—networks, norms, and trust—that enable participants to act together more effectively to pursue shared objectives.”
Elites were more cooperative back then. Since 80ies the elites are into competition & individualism of Ayn-Rand and Hayek (?) Social Darwinism where ‘Greed is right, Greed works’;
Says crash of 2008 is similar to crash of the 1870ies.</p>
<p>there is ‘kin selection’ - altruism is good for the species so this trait is beneficial for survival/evolution selects it in.
Problem: doesn’t work ‘rational agents’ would always prefer to defect from battle – the potential personal cost is too high.</p>
<p>Dawkins ‘selfish gene’: evolution selects for self interest of passing your genes on
still altruistic behavior can be evolutionary advantage if it helps to get close relatives through (kin selection)
‘tragedy of commons’ - everyone wins if the commons is preserved, but your own benefit is to grab as much as possible (this destroys the commons).
Dawkins says reciprocal altruism fixes this (handy to do good – and to get a favor in return)</p>
<p>:: still all this does not explain behavior of unrelated individuals; also ‘kin selection’ and ‘reciprocal altruism’ work adversely in very large societies – here they take the form of nepotism and cronyism.
Says that ‘ultra society’ is fundamentally different from small scale tribe and that it came out of cultural evolution.</p>
<p>Sport teams: conflict of either help your peers (group goal of winning) or playing it out on yourself (goal of higher standing in group)
‘multilevel selection’ deals with such conflicting situations.</p>
<p>Should high performing team mates get higher pay? Cultural thing: Nordic countries (and Australia) are egalitarian so all team members get the same pay; Americans don’t do that.
Too unequal teams have great players – but the rest slack off (and in group competition makes you play against your own team), so they loose games (players work harder in more equal teams)
(says they still do it because stars draw crowds, and wealthy backers overestimate the role of stars)
Same for business.</p>
<p>Multilevel selection says: in-group competition destroys cooperation/teams; competition between group creates cooperation/teams.
That’s why European soccer is relegating lowest performing teams to the lower league – rank and yank applied to groups.</p>
<p>enter the very vague field of Cultural evolution – how do cultures progress?
One problem – they classify clear stages of development (whereas different cultures took different paths).
Another problem – how is cultural information represented in the brain? What is the subject of the study? - Don’t know.
What about concrete features - is trust of strangers (generalized trust) a cultural feature? Banfield: south Italy they don’t trust strangers - this is used to explain their poverty.
However attitudes seem to change over time – evolve.</p>
<p>Price equation is supposed to explain how conflicting traits balance out – but i did not get it.
(free riders are good for individual, but bad for society, altruism is bad for individual but good for society, anyway you need both of them in the long run)</p>
<p>Says that the equation does not care if culture is transmitted by education or by genes or by imitation.
Where did culture come from? Says that extremely variable/cyclic culture + long lifespan of Humans forced us to do behavioral adaptation (learning) instead of adapting the organism to changing conditions.</p>
<p>Also if variation is allowed within an organization then the best approach can be selected (also follows from price equation). In-group competition must be reduced for a team to win.</p>
<hr />
<p>Throwing stones is the first weapon of humans – homo erectus has shoulders for doing so (other apes are not very good at throwing) => Now can get bone marrow and meat; energy requirement for brains.
Gintin, van Schaik say that projectile weapons (spears) made us human; Wranham says fire was important (digestive track shrank - more energy from cooked food enabled larger brains; family formed around fire…)
Early Human forager societies were more egalitarian than Chimps or Gorillas (here the alpha male/beta beats up all the others and get all females, but there can be coalitions against alpha)
Says that projectile weapons were equalizer and restraining the alpha males (easy to surprise the enemy with projectiles; can be used as collective attack; skill more important than physical strength)
Also Humans do coalitions of ten or more individuals - tools and language help. (Chimps have two or maximum three); Leaders had to persuade and build consensus (not by force) = free riders get isolated and punished (moralistic punishment)
social/political skills become important so that women can gain influence.
competition between groups selects for those who developed corporation.</p>
<p>====
Enga island of New Guinea – had people still in stone age; here guys are busy with war - (this kills 35% of them).
War is ceremonial business – not many get killed in each battle – but over time it accumulates (that’s why they got the notion of ‘primitive war’ - misconception)
Now all this fighting did not result in any cultural evolution – the tribes stay the same (surprise) - here everyone is suspicious of everybody else (outsiders could be spies)
all Enga tribes had the same culture - both victors and those defeated; no evolution. (in other forest ares of Indonesia there is also a lot of linguistic diversity – means that no group got extinct; no intense competition; flatland of Eurasia is different – languages spread out)</p>
<p>Ian Morris: wars between societies create competition and larger societies; in-group war (same society) does not create anything -
Author says inconclusive wars do not select anyone out; Destruction of cultural traits means to ‘select out’ - can be genocide or ethnocide (assimilation of culture) or voluntary decision of a culture to change.</p>
<p>Funny thing is that both anthropologists and military historians get the nature of warfare wrong – because they don’t get asymmetric warfare or because they don’t get the details right (important thing is mobility & strike at a distance)
Western way of war’ often glorifies short range combat (bayonets in WW1; hoplites in Greece) ; but mounted horsemen killing at a distance with bows are always more efficient (Parthian armored cavalry against Roman testudo (turtle formation); English archers at Agincourt; cannons in hundred year war)</p>
<p>========</p>
<p>Says early agricultural states were much more unequal than hunter gatherers – the leader would be deified and get total subordination / slavery is wide spread; women were much more suppressed.
Example: Cook expedition was amazed how despotic Hawaiian chiefs were towards their subjects (included human sacrifice; deification of the ruler; etc)
Bruce Trigger says all early societies were despotic, with human sacrifice, some with deified leader.
Says commoners did not like their ‘god rulers’ too much in kind – songs preserved from the Shi Jing (500BC) say so.
seems to take several thousand years from adoption of agriculture until you get archaic states with deified rulers.
Examples of agricultural society that stayed small farm and did not get hierarchies – New Guinea, areas in South America, Africa
Says size is important: once farming community gets larger than 10k people then it get hierarchical; large groups out competed the small ones (‘increasing returns to scale’)
Wittvogel: large scale irrigation needed lot of work and required large groups and centralized administration – that became despotic.
Problem: Russia for example needs no irrigation – but has despotism.
Problem: Mesopotamia – irrigation systems were constructed on village level – don’t need a centralized bureaucracy.
Other advantages: long distance trade; food reserves to cope with famines.
Leeuw: need large groups to solve complex problems + specialized knowledge. Larger groups create more problems - require even larger groups.
Another theory: theocracies drove creation of states (me – problem: in Mesopotamia king and court were parallel structures)
Example: Harrison: in Polynesia sometimes a clan tries to concentrate both priest + ruler position (normally these are separate); sometimes they manage.
Still does not explain how these groups took hold – people would get rid of them (egalitarian impulse).
Could ‘economies of scale’ persuade people that despotic rule is better?
Oppenheimer: One reason is force - people internalized that they had to bow, and did so + religious propaganda. Did stratification arise spontaneously out of differentiation by wealth?
Problem: Egypt old kingdom was not unified by force – unification came from within.
Says the advantage is warfare – Lanchester’s square law – twice the size of the army gives you twofold advantage (for raged weapons that allow simultaneous firing on flat terrain); this translates into selection pressure for larger polities that can field larger armies.
Robert Michel: Iron law of oligarchy – all forms of organization end up as oligarchies.
Robert Carneiro: temporary war chiefs would stay as permanent maximum leader => permanent chiefdom (were this went wrong: Arminius and Marabous got killed when they tried to extend their war chief position; Julius Cesar got killed by fans of the Roman Republic)
Turchin says it was most likely a clique around the temporary war chief; as there is a lot of institutionalized opposition against usurpers – Roman Republic as an example of institutions that tried to avert usurpers.
Still Octavian succeeded to become emperor – because he was modest, got popular support and people were tired of the civil wars.
Same for Germanic society: 100BC – bands of marauding warriors; 1st century – alliances of tribes with centralized command (Arminius); 300BC – permanent confederations Ostrogoths, Visigoths, Merovingians; etc.</p>
<p>How did we get states? Bottom up theory: agriculture allowed to sustain cities + division of labor; surpluses could feed priests and rulers. Now some theories favor economy, some religion, some war aspects.
Problem: Goebeki Teppe and Poverty Point came before agriculture; whereas agriculture is supposed to be prerequisite.
Problem: small communities do food sharing - but hoarding of surpluses would not be ethical.
Problem: with agriculture people were more sick - (pathogens from cattle, field toil is hard, food is worse - from bones of remains )</p>
<p>Turchin explains this as follows: (Pleistocene – last ice age) 12800-11800 – climate became really dry; plants became scarce -> conflict between tribes spiked; created pressure to make tribes bigger.
Now need glue to keep society together – act of building stone henge and megaliths did the job (focus of collective action). Group selection for best fighters made them swallow the disadvantages of agriculture + everyone had to adopt it (came with institutions, property rights, etc).
Property rights -> wealth differential (Mathew principle)
Problems: communities have institutions for wealth redistribution – rich men are expected to donate significant means to festivals – to gain respect; greedy hording is not honorable. (that is called Big Man society)
Says upstart could not keep temporary war chief privileges – but god kings could keep them (by combining position of king and priest/god)
God kings were more effective warrior states - because ruler was not restrained by moral obligations (? internal cohesion problems?)</p>
<p>The tide turned during the axial age: Mega empires have been formed (Achimenid empire; Maruyan empire; Han dynasty) - tens of millions subjects.
Ashoka – founder of Buddhism comes of as a king that is concerned for the well being of his subjects! Also propaganda is not as bombastic as it used to be for the rulers of old. Same independently occurs in Judaism – an unrelated religions (Judaism).
Says archaic states were fragile in time of peace - due to restiveness/internal upheaval caused by grave inequalities; for stability you need legitimate authority (hierarchy) ; not ‘dominance’ (R. Bellah)
Axial age sees social criticism – by Prophets (Amos, Samuel); Buddha; Taoists/Confucians; Plato (?)</p>
<p>Trigger: Mil-tech advance of horse (needs saddle and bridle)/chariots – later mounted cavalry/ + composite bows: both were pushed by nomads (1000BC) This intensified warfare.
Iron use – was more abundant than Copper/tin; now mobile warfare was in.
First ones to integrate cavalry + infantry + siege engines were the Achaemenids/Medes; Jaspers also thought that axial age was prompted because Nomads caused intensification of warfare.
Greek Hoplites (armed infantry with shields) + phalanx formation could counter the mounted cavalry + archers; but it needed larger states to field huge armies and defend borders - this countered the disadvantage in mobility;
needed large armies of commoners so it could no longer subject them by force alone (also had to give commoners weapons) ; elites became more receptive to message of prophets.</p>
<p>Example: Rome at 500BC - the plebeians were hoplite warriors and went on strike (secessions); got concessions of more equality + share in the spoils in return for cooperation (? why didn’t rome get an axial age religion back in the Republic; what about Stoicism ?)</p>
<p>Here come axial age empires (Achemenid Persia, Rome, Han dynasty, Mauryan (? How could fragmented Greece do without a centralized one ?)
Achaemenid empire had huge problems with stability - problem of uprisings. says Axial age religions became important to hold things togather (? Christianity came in towards the end of western Rome - Gibbons says it did them in ?)
Still Zoroastrism; Budhism and Confucianism were all attempts to hold their empire togather.
Mechanism is that large empires have multiple layered levels of idenity - one of these identities is the state religion of an empire; and that is a prerequisite for trust in large societies were people have to deal with strangers.
Cooperation is higher in societies that believe in a big God (Norenzayan) who 1) knows your thoughts 2) evaluates if you are virtuous 3) punishes for transgression; fear of a bad afterlife will cause you to be a better cooperator (in large societies).
Nonbelievers will not be trusted - don’t get any deals; its very costly. Rulers have to take that into account (in US an atheist candidate for president will not be trusted).</p>
<p>======
Summary:</p>
<p>if you measure eqauality on a graph over time then it follows a z curve - Humanoid/apes are very unequal with dominating alpha; early humans developed ‘equalizers’ and institutions that stood for it - also selected for the tribes that were able to achieve collective action;
then advances of warfare turned larger batallions into the winning argument, so god-kings turned tyrants and took it all;
then the model did not scale - further intensification made it important to have mega-empires that needed a more equal approach (axial age).</p>
<p>Violence took a spike in the pleistocene (last ice age) - climate took turn for the worse - tribes fought over territory; groups that succeded to develop collective action won (team building done by doing megalyths)
Later development saw decrease in violence (percentage wise always a decrease of casualties (? except for very bad wars that cut down half a significant portion of the population - ask the war widows in the USSR ?))
Soedberg: half of deaths in foragging bands was due to interpersonal violence/’crime’.</p>
<p>Pinker: reasons for decline in violence / monopoly of state on force - inhibits revenge impulse / Comerce - everybody wins from the exchange, no profit in dehumanising the ‘enemy’ / Feminization - violence no longer glorified, guys don’t form gangs / cosmopolitanism - increases sympathy for the other one</p>
<p>Problem: many reasons, here he searches for unifying causes – cooperation evolved, institutions & values got formed that diminish strive: you can have the best judges, but if everyone says that bribes are OK then it does not work.
Problem: evolutionary psychology of Pinker focuses on motives of individuals, so he ignores the factor of cultural change over time and how states have arisen – cultural evolution does not have this problem because that is what it studies.
Also helping your friends is OK in small societies – in state structures it takes the form of nepotism/cronyism.
Problem: commerce not always zero sum game, can be very unequal – like in colonialism;</p>
<p>In Mega empires the percentage wise number of casualties was falling – but importance of the rest of society to the war effort increased.
Post WW-2 competition was who has the better economy and propaganda (and the biggest bombs…)</p>
<p>Says cultural traits get adopted if they are of advantage: monogamy – it spread by imitation to Asian societies, but it also makes society more competitive: more equality among men & women, less in-group competition, less crime (less single males), more investment into offspring, more prosperous society.</p>
<p>Prevention of failed states – as cooperation breaks down, so do states. finding out why cooperation happens will help to preserve it + effective policy recommendations.</p>
Rust - the new kid in town
2016-05-06T00:00:00+00:00
http://mosermichael.github.com//all/blog/2016/05/06/hello-rust
["", "blog"] | <h2 id="rust---the-new-kid-in-town">Rust - the new kid in town</h2>
<p class="publish_date">
6 February 2016
</p>
<p>Rust is a promising new programming language</p>
<ul>
<li>Rust is similar to C** in terms of runtime costs: memory management is explicit, there is no automatic garbage collector.</li>
<li>The main design philosophy (like in C**) is to provide higher level language features that are checked at compile time with no additional runtime cost, all in order to keep runtime complexity as low as possible.</li>
<li>Rust tries to provide memory safety: all heap allocated memory has its owner (owning variable), memory of heap allocated data is freed once the owner of this memory goes out of scope. Lifetime of references is enforced by the compiler, a program that passes compilation is supposed to be free of access to freed/invalid memory. This is a really big improvement over C**.</li>
<li>In terms of syntax and type system: Rust tries to favor a more functional style of programming. It has closures, a function can return a closure as return value (!); it also has variant enum (a type that can assume any one of a given list of types - that is similar to haskell data definitions) and pattern matching switch statements. (The Rust Option type/error handling is very similar to the Haskell Maybe monad). Rust also has local type inference (like C++11 with auto, go - all newer languages do have this feature).</li>
<li>Object oriented programming: A structure can have member functions; all structure fields are ‘public’ (no data hiding). Rust has interface inheritance (you can implement interfaces), but no implementation inheritance (you can’t derive from another structure). Rust has templates (unlike Go).</li>
<li>Modularity</li>
<li>macros: Rust has quite complex macros, the macro can match entities over the syntax tree and manipulate them (interesting how they will fix slow compilation time while keeping this feature)</li>
</ul>
<p>How does Rust compare to C** in terms of language complexity? For C** a lot of the complexity comes from the requirement of backward compatibility with C, Rust does not have such a problem. Now Rust does introduce a lot of features that are lacking in C**; these features do add complexity and it doesn’t make Rust a simple language. However the promise of safety from memory errors makes it all seem worth the cost.</p>
<p>It is interesting to note that statically typed languages seem to be all the rage these days. Scripting languages were popular in the 90ies and 00’s, when the internet/web experienced massive growth and tools that made it easy to develop for the web platform got a lot of attention. Now the party is sort of over as there is no longer the promise of massive growth; also CPUs are not getting much faster as they used to do - the focus now is more efficient use of existing computing resources; the changed realities call for statically typed compiled languages that target native code. Of course an added focus is on security it is too easy to introduce subtle security bugs with C and C**. Rust tries to fit into this niche lets see how fast we will see wide spread adoption.</p>
<p>It will take some time until Rust becomes a mainstream language, it took quite some for C** and Java to become significant players (more than a decade); it takes a lot of time for the language to mature, a ‘big’ language also needs a significant pool of trained programmers training of a large number of programmers takes time and is very costly.</p>
<p>Also I suspect that it will take quite a while to master this language ‘efficiently’/’effectively’ (just as with C**).</p>
<hr />
<p>My notes on the <a href="https://doc.rust-lang.org/book/">Rust book</a></p>
<p>Variables. values / simple types</p>
<ul>
<li>builtin types (like stdint.h) i8,i16,i32,i64 signed integers, u8, u16, u32, u64 unsigned integers; isize, usize equivalents for ssize_t, size_t (in C). f32, f64 floating point types.</li>
</ul>
<blockquote>
<code><pre>
let x = 5
</pre></code>
</blockquote>
<ul>
<li>
<p>x is variable on the stack (by default).</p>
</li>
<li>does type inference; x is immutable (constant) variable; can’t assign x = 7</li>
<li>the full declaration (with type) is</li>
</ul>
<blockquote>
<code><pre>
let x : i32 = 5
</pre></code>
</blockquote>
<ul>
<li>also: you must assign a value to a variable at variable declaration</li>
</ul>
<blockquote>
<code><pre>
let x : 32 // a syntax error
</pre></code>
</blockquote>
<ul>
<li>if you want variable allocated on the heap then</li>
</ul>
<blockquote>
<code><pre>
let x = Box::new(5); // this creates instance of Box<i32> - it points to i32 value allocated on heap (Box<i32> - instantiate a template of type Box that works with i32 (Box is in standard library)
</pre></code>
</blockquote>
<ul>
<li>also: variable lifetime is limited to its scope</li>
</ul>
<blockquote>
<code><pre>
{
let x = 5
}
println!(" {} ", x ) // can't do that ; x is used out of its scope so x is no longer visible.''
</pre></code>
</blockquote>
<p>also a variable can be redefined (shadowed) in its nested scope, but only the last scope is valid</p>
<blockquote>
<code><pre>
let x = 5
{
let x = 7 // x is now 7
}
// x is 5 again
let mut x = 5
x = 7
</pre></code>
</blockquote>
<p>-</p>
<ul>
<li>that’s possible because x is now mutable. (by default a declaration is immutable (const))</li>
</ul>
<blockquote>
<code><pre>
let mut x = 5
let y = & mut x
</pre></code>
</blockquote>
<ul>
<li>you can’t assign something else to y because it is not mutable (it is const) ; but you can modify the referenced value and change x !</li>
</ul>
<blockquote>
<code><pre>
* y = 7 // special syntax for changing mutable refs (dereferencing mutable references)
</pre></code>
</blockquote>
<ul>
<li>now the value of x is 7</li>
</ul>
<p>Functions:</p>
<p>rust program must have a main function (like C); every statement must be in a function</p>
<blockquote>
<code><pre>
fn main()
{
}
</pre></code>
</blockquote>
<ul>
<li>functions : -> <type> denotes the return value.</li>
</ul>
<blockquote>
<code><pre>
fn add( x : i32 ) -> i32
{
x + 1 // block is an expression
// so last statement in block is the return value (but if the last statement ends in ;
// (semicolon) then syntax error because there is no return value !
}
</pre></code>
</blockquote>
<ul>
<li>can also do return to explicitly return a value</li>
</ul>
<blockquote>
<code><pre>
fn add( x : 32 ) -> i32
{
return x + 1 // same as previous but with explicit return
}
</pre></code>
</blockquote>
<ul>
<li>closures (anonymous functions)</li>
</ul>
<blockquote>
<code><pre>
let num = 5
let plus_num = |x : i32 | { x + num } // | <parameter list > | <body> ; also references num another binding.
</pre></code>
</blockquote>
<ul>
<li>closers can mutate captured variables (but they have to be marked as mutable)</li>
</ul>
<blockquote>
<code><pre>
let mut num = 5;
let mut add_num = | x | num += x; // the closure is mutable and it modifies a mutable binding num
add_num(5); // after the call the value of num has changed to 10
</pre></code>
</blockquote>
<p>but the closure can be made to copy the referenced captured variable (move capture)</p>
<blockquote>
<code><pre>
let num = 5
let mut add_num = move |x | num += x
add_num(5) // after the call num is still 5 ! - the move capture made a copy of captured variable num
</pre></code>
</blockquote>
<p>.
data / more complex data structures</p>
<ul>
<li>arrays fixed size arrays</li>
</ul>
<blockquote>
<code><pre>
let v = [1,2,3];
println!( "the first one {} number of entries {}", vec[0], vec.len() ); // prinln! Is a macro call, first {} puts in the string for second param, etc.
let n = [ 1 ; 20 ]; // twenty entries of value 1
</pre></code>
</blockquote>
<ul>
<li>vectors dynamic vectors allocated on heap. (vec! Is a macro call generated code constructs a vector and fills in the specified data)</li>
</ul>
<blockquote>
<code><pre>
let v = vec![ 1,2, 3 ]
</pre></code>
</blockquote>
<ul>
<li>out of bound access will result in panic (program abort)</li>
</ul>
<blockquote>
<code><pre>
println!("will panic ! {}" , v[4] )
</pre></code>
</blockquote>
<ul>
<li>fixed size string (unicode string UTF8) (string constants on initialized data section)</li>
</ul>
<blockquote>
<code><pre>
let a = "can span
multiple lines"
</pre></code>
</blockquote>
<ul>
<li>can’t access n’th element directly but have to convert to chars</li>
</ul>
<blockquote>
<code><pre>
let secondLetter = a.chars().nth(1)
</pre></code>
</blockquote>
<ul>
<li>or convert to array of bytes</li>
</ul>
<blockquote>
<code><pre>
let secondByte = a.as_bytes().
</pre></code>
</blockquote>
<ul>
<li>dynamic strings (on Heap) that can be concatenated type String</li>
</ul>
<blockquote>
<code><pre>
let hello = hello.to_string() // to_string turns str to String.
let world = world // of type str (fixed size string literal)
let hw = hello + world // note that second variable is fixed size literal !
Dereferencing of String gets you a str (because operator * is overloaded for String)
</pre></code>
</blockquote>
<ul>
<li>structures</li>
</ul>
<blockquote>
<code><pre>
struct Point {
x : i32,
y : i32,
}
let point = Point { y : 5 , x ; 4 } // any order of the fields can do as long as all fields (name : value pairs) are specified
println!(" x {} y {}", point.x, point.y )
</pre></code>
</blockquote>
<ul>
<li>tuples: a struct with fields/members that have no name</li>
</ul>
<blockquote>
<code><pre>
struct IntPair { i32, i32 }
let IntPair tuple = IntPair( 15, 20 )
</pre></code>
</blockquote>
<ul>
<li>
<p>can access values as tuple.0 first member of value; tuple.1 second member of value , …</p>
</li>
<li>
<p>or more tricky by means of pattern matching let (assignment)</p>
</li>
</ul>
<blockquote>
<code><pre>
let IntPair encoded = IntPair( 15, 20 )
let IntPair( x, y ) = encoded // assigning with pattern matching
println!( "x {} y {} " , x, y );
</pre></code>
</blockquote>
<ul>
<li>interfaces are called traits</li>
</ul>
<blockquote>
<code><pre>
trait Printable
{
fn print( &self );
}
struct Circle {
x : f64,
y : f64,
radius : f64
}
impl Printable for Circle {
fn print( &self )
{
println!(" x {} y {} ", self.x, self.y )
}
}
</pre></code>
</blockquote>
<ul>
<li>traits can inherit (the inherited interface adds additional methods)</li>
</ul>
<blockquote>
<code><pre>
trait Foo {
fn foo(&self);
}
trait FooFoo { // FooFoo has both foo and bar.
fn bar(&self);
}
</pre></code>
</blockquote>
<p>if a struct implements FooFoo then it has to implement both Foo and FooFoo traits !</p>
<p>template types can be required to support a trait/interface</p>
<blockquote>
<code><pre>
fn print_me<T : Printable>( x : T ) { x.print() } // T : Printable == require type T to implement trait Printable
</pre></code>
</blockquote>
<ul>
<li>of you can require that a template type supports multiple interfaces</li>
</ul>
<blockquote>
<code><pre>
fn printMe< T : Printable + Debug> ( x : T ) // requires that x supports both Printable and Debug trait
</pre></code>
</blockquote>
<ul>
<li>or equivalent form</li>
</ul>
<blockquote>
<code><pre>
fn PrintMe<T> ( x : T) where T : Printable + Debug
</pre></code>
</blockquote>
<ul>
<li>that is supposed to be more convenient with multiple template types that have several requirements.</li>
</ul>
<blockquote>
<code><pre>
fn PrintMe<T : Printable, U : Printable + Debug> ( x: T, y: U)
fn PrintMe<T,U> (x : T, y: U ) where T : Printable, U : Printable + Debug
</pre></code>
</blockquote>
<ul>
<li>with generic structure implementation you can require the template type to support a given interfaces</li>
</ul>
<blockquote>
<code><pre>
struct Rectangle<T>
{
x : T,
y : T,
width : T,
height : T,
}
impl<T : PartialEq> Rectable<T> { // require T to support Partial Eq (? why is this requirement not in the type definition ?)
fn is_square(&self) -> bool {
self.width == self.height // if T supports ParialEq then it has a overloaded == operator.
}
//
</pre></code>
</blockquote>
<ul>
<li>Object Oriented p. with structs (member functions and constructors)</li>
</ul>
<blockquote>
<code><pre>
struct Circle {
x : f64,
y : f64,
radius : f64
}
impl Circle { // member functions are added in one or more impl blocks.
fn new( x : f64, y f64, radius : f64) // new is a static method it does not require this/self.
{
return Circle{ x : x, y : y, radius : radius );
}
f64 x(&self) -> f64 { return x.x } // member function self reference
Circle x(mut &xelf, x : f64 } -> mut & Circle{ self.x = x; return x; } // mutable reference to self. Returning ref to self allows you to chain calls.
}
</pre></code>
</blockquote>
<ul>
<li>templates/generics</li>
</ul>
<blockquote>
<code><pre>
enum Option<T> {
Some(T),
None
}
- let x = Option(5) // type x inferred to be Option<i32>
let y = Option(5.0) // type y inferred to be Option<f32>
</pre></code>
</blockquote>
<ul>
<li>function templates</li>
</ul>
<blockquote>
<code><pre>
fn do_it<T, U> ( T : x, Y : y)
</pre></code>
</blockquote>
<ul>
<li>structs with templates</li>
</ul>
<blockquote>
<code><pre>
struct Point<T> {
T : x,
T : y,
}
impl<T> Point<T> {
fn (&mut self, T x } -> Point<T> { self.x = x; return x }
}
</pre></code>
</blockquote>
<ul>
<li>
<p>overriding stuff:</p>
</li>
<li>each type can have a destructor that is called when an instance of the type goes out of scope.</li>
<li>you get that by implementing the Drop interface for the type.</li>
</ul>
<blockquote>
<code><pre>
struct HasDTor {
int a;
}
impl Drop for HasDTor {
fn drop(&mut self) {
println!("calling the dtor ");
}
}
</pre></code>
</blockquote>
<p>control structures:</p>
<p>rust has its own switch statement (called pattern)</p>
<blockquote>
<code><pre>
match x {
1 => println!( "1\n" ),
2 | 3 => println!( "2 or 3\n" ),
4 .. 7 => println!( "4 or 5 or 6 or 7 a range as selector\n" ),
e @ 8 .. 12 => println!( "for range you can get value of the match {} \n", e )
_ => println!("something else default\n" ), // _ => matches all other values
}
</pre></code>
</blockquote>
<p>this match thing most cover all values ; so without _ => it gives error that not all values are covered by this switch.</p>
<ul>
<li>it can also retrieve structure values</li>
</ul>
<blockquote>
<code><pre>
struct Point { x : i32, y : i32 }
p = new Point{ x: 1 , y : 2 }
match p {
Point{ x, y } => println!("x={} y={}\n", x, y ),
}
</pre></code>
</blockquote>
<ul>
<li>
<p>more usefully it is used to handle enum values each case is handling a possible type of the enum</p>
</li>
<li>
<p>it also works on enums (a union where each field is of different type, but which one of them is in effect is chosen at assignment)</p>
</li>
</ul>
<blockquote>
<code><pre>
enum Message {
MoveIt { x : i32, y : 32 }, // struct with two fields
ChangeColor( i32, i32, i32 ) // tuple with three fields
};
let m : Message = Message::ChangeColor(1,2,3) // we determine that m holds a ChangeColor tuple
</pre></code>
</blockquote>
<ul>
<li>each case deconstructs the type value (fields are extracted in right hand side)</li>
</ul>
<blockquote>
<code><pre>
match m {
MoveIt( x, y ) => { println!( "x{} y {}" , x, y ) // this case handles MoveIt type
ChangeColor( x, y, z ) => println( "x {} y {} z {}", x, y, z)
}
</pre></code>
</blockquote>
<ul>
<li>structure fields can’t be declared mutable its the variable (binding) of type point can be mutable or not.</li>
</ul>
<blockquote>
<code><pre>
struct Point
{
x : i32,
y : y32,
}
mut p = new Point { x: 5, y : 7 }
p.x = 11
pp = new Point { x : 1, y : 2 }
pp.x =22 // compilation error:
</pre></code>
</blockquote>
<p>but no, you can emulate fields that can be modified in non mutable bindings !!! (why?)</p>
<blockquote>
<code><pre>
use std::cell::Cell
struct Point
{
x : i32,
y : Cell<i32>
}
let p = new Point( x: 5, y : Cell::new(6) }
p.y.set(8) // now value of y is 8.
</pre></code>
</blockquote>
<ul>
<li>Enums (not your father’s C enum here each member is a type, so it is a union of types, but the compiler knows that one specific type has been chosen as the value and enforces it accordingly)</li>
</ul>
<blockquote>
<code><pre>
enum Message {
MoveIt { x : i32, y : 32 }, // struct with two fields
ChangeColor( i32, i32, i32 ) // tuple with three fields
};
let x : Message = Message::MoveIt{ x : 10, y : 20 } // constructs an instance of message and sets its implementation type to MoveIt. Value x can't be treated as ChangeColor !!!
</pre></code>
</blockquote>
<ul>
<li>
<p>messages are processed with match keyword (see under control structures)</p>
</li>
<li>
<p>member functions (structures and enums can have member functions_</p>
</li>
</ul>
<blockquote>
<code><pre>
impl Point
{
// each new member function can be in its own impl block (or several grouped together)
fn distance(&self, Point other) -> f64
{
(self.x other.x) * (self.x other.x) + (self.y other.y) * (self.y other.y)
}
}
impl Point
{
fn move(& mut self, Point other) -> & mut Point // mutable this if you want to modify the object !
// return value of mutable self so that you can chain methods
// let &mut pt = Point{ x : 0 , y : 0 }
// pt.move(Point{ x: 10 , y: 20 } ).distance( Point { x : 100, y: 100 } )
{
self.x += other.x;
self.y += other.y;
self
}
}
</pre></code>
</blockquote>
<ul>
<li>generics: syntax is similar to C** and Java templates.</li>
</ul>
<blockquote>
<code><pre>
struct Point<T> {
x : T,
y : T
}
</pre></code>
</blockquote>
<ul>
<li>one can also parametrize functions:</li>
</ul>
<blockquote>
<code><pre>
fn two_args<T,U> ( x : T, y : U )
</pre></code>
</blockquote>
<p>Memory management / Ownership</p>
<ul>
<li>
<p>the main ‘killer’ feature of Rust; makes sure that only one variable owns the data (one binding)</p>
</li>
<li>
<p>Move types : this means that assigning the value moves ownership</p>
</li>
</ul>
<blockquote>
<code><pre>
let v = vec![ 1, 2, 3]
let u = v // for vector this moves ownership to u; can't use v afterwards
println!("{}", v[0]) // compiler error: acessing moved value !!!
</pre></code>
</blockquote>
<p>move semantics are the default: but if type implements Copy trait then it is copied by value (like i32 etc)</p>
<blockquote>
<code><pre>
let n = 1
let m = n
println!("still can use n {} because n was copied not moved", n )
</pre></code>
</blockquote>
<ul>
<li>
<p>also passing a vector to a function also moves ownership from caller of function to function implementation (like assignment); so use references for every function argument that is not easily copied.</p>
</li>
<li>References: (making a reference is called to ‘borrow’)</li>
<li>Rule: there can be more than one ‘const’ reference to a value;</li>
</ul>
<blockquote>
<code><pre>
let num = 32
let refToNum = &num
let anotherRefToNum = &num
println("get a value from ref {}" , *refToNum ) // dereference
</pre></code>
</blockquote>
<ul>
<li>Rule: there can be one mutable reference (where you can change the referred to value)</li>
</ul>
<blockquote>
<code><pre>
let mut num = 32 /
{
let & mutRef = & mut num // & mut creates mutable reference, referred to value must be mutable to begin with !
*mutRef += 1 // now value of num is 33, you need * only with Mutable references. !!!
}
println!( "the value is now {}", num ); // can create new 'const' reference only when there is no active mutable reference !!!
</pre></code>
</blockquote>
<p>Rule: you can’t make a new ‘const’ reference while a mutable reference exists/
(q: what about dereferencing of existing ‘const’ reference while mutable reference exists?)
that rule prevents data ‘races’ access of value while another part of the program may modify the value.</p>
<ul>
<li>
<p>rational: supposed to prevent iterator invalidation in a loop each iteration creates new const ref; but modification of collection (owner) involves holding mutable reference.</p>
</li>
<li>
<p>Lifetimes: goal: prevent access of freed memory via reference.</p>
</li>
</ul>
<blockquote>
<code><pre>
fn two_refs_as_args( x : & int32, y &int32 ) -> & int32 // does not compile !!!!@
</pre></code>
</blockquote>
<ul>
<li>
<p>now the function could sometimes the first reference with one lifetime, and sometimes the second one (ambiguity); so it is impossible to infer the lifetime of the returned reference for the general case !</p>
</li>
<li>also if the caller of the function is in a different package then the compiler does not always have the source code of the function;</li>
<li>therefore need to specify the lifetime of each ref (unless the compiler can’t infer it for very simple cases as function with one argument)</li>
</ul>
<blockquote>
<code><pre>
fn two_refs_as_arguments<'a, 'b> ( x : & 'a int32 , x : & 'b int32 ) -> int32 & 'a
</pre></code>
</blockquote>
<ul>
<li>lifetimes are specified as ‘a and ‘b (both are part of the type signature of the function). They are given as function ; lifetime is given before before type.’’</li>
</ul>
<p>Same applies to structures that hold two references as member fields : a member function of the structure might sometimes return the first structure field (reference) and sometimes the second one, in this case it is impossible to infer the lifetime of the return value.</p>
<blockquote>
<code><pre>
struct Foo<'a,'b> {
x : int32 'a,
y : in32 'b,
}
impl Foo<'a,'b> Foo<'a> {
fn x(&self) -> & 'a i32 { return self.x }
fn y(&self) -> & 'b i32 { return self.y }
}
</pre></code>
</blockquote>
On Parenting
2015-12-20T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2015/12/20/on-parenting
["", "blogg"] | <h2 id="on-parenting">On Parenting</h2>
<p class="publish_date">
20 Dec 2015
</p>
<p>“The Highly Sensitive Child: Helping Our Children Thrive When the World Overwhelms Them” by Elaine Aron</p>
<p>A very interesting and wise book about how to raise a highly sensitive child; it explains how to see the world with his/her eyes; it teaches empathy and confidence in the abilities of the child. A great book on how to be a better parent. (Purchase at the <a href="http://hsperson.com/store/bookstore/">author’s site</a> or at <a href="http://www.amazon.com/The-Highly-Sensitive-Child-Overwhelms/dp/0767908724">Amazon</a> )</p>
<p>I think that it is not just about kids who are highly sensitive by nature; everybody who happens to grow up as part of a minority, or as an outsider is in a similar situation; Being an outsider places the individual in a situation that forces him/her to notice finer social difference (these can be safely ignored if you are part of the majority).</p>
<p>Also I think that the general approach of trying to understand, respect and engage your kid is of great benefit to all kids; in a way we are all highly sensitive.</p>
<p>Here are my notes on two key chapters; there is much to learn from every chapter! I can only recommend to buy this book and to work through the whole of it. I am very thankful for this book.</p>
<iframe width="420" height="315" src="https://www.youtube.com/embed/ztVaqZajq-I" frameborder="0" allowfullscreen=""></iframe>
<hr />
<p>Chapter 1</p>
<ul>
<li>highly sensitive child (HSC)
<ul>
<li>notice more (finer perception in one or multiple senses) and reflect more deeply (more thorough/intensive processing of information); think more, but often processing is on a sub/unconscious level</li>
<li>Are easily overwhelmed by lots of information; affected more deeply by stimulus - overstimulation (too much noise, too long family reunions) results in extreme reactions: tantrums + meltdowns (the child is trying to say here: ‘I can’t take it anymore’), persistently avoid negative stimulus, perfectionism as a way to overcompensate. Need more downtime (time to relax) - to absorb stimuli.</li>
<li>more affected by pain; more sensitive immune system (often sick); often faster reflexes</li>
</ul>
</li>
<li>15-20% are ‘highly sensitive’ (same number with boys and girls) That’s too much to be ‘abnormal’ - the success of upbringing depends on the environment: support from the family. Says that among animals there are highly sensitive and lower sensitive animals - both types compliment each other (in terms of evolution its not a trait that is weeded out). Large minority of HSC is of benefit - notice dangers earlier, help others; creativity.</li>
<li>Says its a spectrum - some are more some are less sensitive; some more tuned to social relations others to external stimulus. There are differences along the following axis of temperament traits:
<ul>
<li>sensory threshold (of things getting noticed) ; which senses? (distinctions that seem small to adult are a really a big deal to HSC !)</li>
<li>activity/energy level</li>
<li>higher intensity of emotional response</li>
<li>rhythmicity (need there order and are predictable)</li>
<li>adaptability: (seem to be poor at adapting - but that is because fear of being overwhelmed, asked to adapt too much); make an effort to fit in at school but let it go at home??</li>
<li>initial reaction; does the child ‘go for it’ when it seems save or does he/she withdraw as initial reaction?</li>
<li>Persistence; tend to be persistent but get upset when things don’t turn out as expected. Does that cause them to quit or go on? How does the kid react to input by others in this case?</li>
<li>Easily distracted</li>
<li>predominant mood: says for HSC mood is more affected by their experiences.</li>
</ul>
</li>
<li>each child is different; nothings wrong with the child - its a gift. ‘when they grow older still, they are often remarkable for their kindness and conscientiousness’; are emotionally healthier as adults (less likely to be depressed or psychotic) - on condition of supportive upbringing (!).</li>
<li>on negative inputs: more likely to become shy, fearful, depressed; That’s why guidance is important.</li>
<li>How comes? It might be genetics; neurology: in HSC the ‘behavioral inhibition’ center is very active (it does pattern matching with past situations) - its in the right hemisphere (the one supposedly responsible for creativity) ; (very different from ADD - here the left hemisphere is more active - HSC often mistreated as ADD because of adverse reaction to stress)</li>
<li>at first the author thought that high sensitivity is ‘highly introverted; but 30% of ‘highly sensitive’ people are extroverts - often the result of supportive upbringing, so that they don’t fear people. (still many HSC prefer two, three close friends to wider circle of communication - saver that way)</li>
</ul>
<p>Parenting- the keys: Self-esteem, shame reduction, wise discipline, knowing how to discuss sensitivity.</p>
<ul>
<li><em>Self esteem</em> - the most important part
<ul>
<li>HSC is often very self-critical (because of perfectionism?) ; also any outside message is always amplified.</li>
<li>The basic attitude is key; it negative then successes are not appreciated and failure is perceived as reinforcing the negative outlook. Too much criticism will lead HSC to conclusion that everything is wrong; (also criticism is perceived more intensely).</li>
<li>School/garden often treats HSC as ‘being wrong’. This leads to shame - and that’s really hard to deal with (because of negative feedback loop/reinforcement)</li>
</ul>
</li>
<li>what are the sources of self-esteem?
<ul>
<li>Unconditional love by parents - a sense of stability is very important (HSC seek stability)</li>
<li>social self-esteem; ability to have and be valued by friends.</li>
<li>Physical self-esteem; value your own appearance and abilities.</li>
<li>Intellectual self esteem - confidence to master learning situations</li>
</ul>
</li>
<li>HSC often is ‘depressed realist’ - will likely to not overestimate his abilities (people with too much self confidence do that). Says that’s because the basic strategy is to ‘get it right the first time’ - (to avoid negative feedback on failure); Also will have guilt feelings for bad thoughts (more aware of unconscious side) - important to teach that thoughts are different from deeds.</li>
<li>Promoting self esteem:
<ul>
<li>lead by example - also your own attitudes are visible to the highly sensitive. (me: don’t talk about yourself in deprecating manner)</li>
<li>Speak admiringly about child’s sensitivity - praise observation skills. (and all positive aspects of HSC); don’t overdo it (to stay credible).</li>
<li>Spend time with the child</li>
<li>Show respect, even in odd situations (I know that you like .. but …)</li>
<li>How to deal with non sensitive people? Teach that many people act impulsively ‘without meaning what they say’ == teach the kid to wear emotional earplugs (me: tell that there people who hurt others as a form of trolling - compensation for their own shortcomings); Teach that different people have different styles. Say that other people may not hear well, so speak louder (but don’t victimize if it does not work)</li>
</ul>
</li>
<li>bring up strengths when the child is confronted by setback/self doubt (past success to match current failure). Don’t overdo praise - praise must stay credible (you have a critic before you). Don’t get into argument - cut it off: its my view. Important:
<ul>
<li>People with low self esteem have a good memory of past setbacks - as if stored under the same label (keyword); so that one failure brings the whole cluster in. The labeling of the kid by its environment is what gets him there.</li>
<li>Target is to have memory organized under other headlines (sports, arts, science, etc); mentioning past successes may offer an alternative viewpoint (that’s why it’s important)- even if it is not accepted by the child.</li>
</ul>
</li>
<li><em>Shame reduction</em>
<ul>
<li>Feeling shame means that the entire self is bad - that leads to withdrawal/submission (as opposed to guilt where the focus is on a particular deed). Worst case scenario: shame/guilt/shyness/anxiety become predominant emotions.</li>
<li>However shame has cultural connotations: In Japan peoples shame/pride depends on evaluations of others (more than in western culture - where one has to be ashamed for feeling shame, with HSC that leads to more shame ..); also one is expected to feel shame - in order to stay engaged with each other. (? HSC feel somehow more familiar in Japan, go figure ?)</li>
</ul>
</li>
<li>different kinds of shame
<ul>
<li>Spontaneous shame for misdeed (not the shame induced by others for being different) can be a form of learning;</li>
<li>Shame induced by others - don’t do that (can have very bad effect to HSC - I am not good, lots of distress)</li>
<li>intense shame - creates hostility as a way to protect against the feeling (directed towards oneself - for HSC mostly against the self ;-( and others)</li>
</ul>
</li>
<li>What to do? Objective is to avoid shame proneness.
<ul>
<li>If you never correct/discipline your child then that also has the message to the child that he/she is beyond redemption (?!) Worst kind of message is that you have given up.</li>
<li>Examine your own words/reactions (got them from upbringing)- don’t shame the child as a habit.</li>
<li>Don’t voice expectations for the future - (not measuring up is a source of shame) ; The message must be: succeed on your terms, its your world, enjoy</li>
<li>Don’t compare with other kids/siblings.</li>
<li>Teasing - kids will not appreciate your habits/sense of humor. Careful.</li>
<li>Don’t impart the impression that the kid is to blame for family problems - that can be very very bad.</li>
</ul>
</li>
<li><em>Discipline</em>: the aim is to teach a moral outlook - what is right and wrong (fear of punishment alone is not enough to regulate behavior)</li>
<li>Theory on origin of moral: Caring relationship between infant and parent - both enjoy sharing and try make peer happy. Still there is stress/need to discipline: child is conflict between
<ul>
<li>trying to have his own way.</li>
<li>Trying to please parent.</li>
</ul>
</li>
<li>Still kid adopts position of parent around age 3 (if all goes well). How to impart values? (General case) kid needs to listen - says that some arousal is good because you can give additional context on social situations (Kids don’t like kids who bite), but don’t scare him/her stiff (counterproductive)!!!</li>
<li>HSC: have it easier to internalize rules, in a way they punish themselves. So just a change in tone is enough to get the child’s attention - more than that means to overdo it.</li>
<li>Make an effort to avoid the need for ‘discipline events’
<ul>
<li>by making your expectations explicit - important that they are age-specific (easy to understand for the given age; even better if they are part of a mutual agreement.</li>
<li>Try to understand the reason + analyze/tackle root causes: (like perceived inequality of treatment between siblings, etc)</li>
<li>Transition points are difficult: It helps if they are part of a routine; humor helps; try to anticipate the crisis in advance; being clear helps (but don’t overextend deadlines).</li>
</ul>
</li>
<li>Discipline (didn’t understand it all - so only part is listed)
<ul>
<li>make an effort to calm the situation - so that the child and parents are not overarroused.; Listen and show empathy</li>
<li>restate standards/expectations - if you reason take care that the child is able to understand your arguments.</li>
<li>Show some alternatives for the future (hit this pillow when you are angry)</li>
<li>again and again: prevent anxiety from backfiring.</li>
</ul>
</li>
<li>Know how to discuss sensitivity - with the child and others
<ul>
<li>
<h2 id="you-have-to-read-it">you have to read it….</h2>
</li>
</ul>
</li>
</ul>
What every pr0grammer should know...
2015-12-11T00:00:00+00:00
http://mosermichael.github.com//all/blog/2015/12/11/wepskn
["", "blog"] | <h2 id="what-every-pr0grammer-should-know">What every pr0grammer should know…</h2>
<p class="publish_date">
11 December 2015
</p>
<p>Maybe the title is supposed to say that the content of the article should have rather have been taught in school… The first article is about what happens when DRAM memory is accessed and how to optimize the programs usage of memory; it explains memory from the hardware up (quite complex hardware that is); part of this article is currently taught in the ‘Computer Organization’ course; but I doubt they go into this level of detail; it took me quite some time to parse this article.</p>
<p>It is funny that back in 2007 the x86 architecture was the only significant player (the focus of the article is on Linux OS on x86;x86_64), while in 2015 things like ARM are becoming more relevant; things do change over time. I guess some details might be outdated by now, but on the whole it will stay relevant for quite some time:</p>
<p>What Every Programmer Should Know About Memory by Ulrich Drepper ( “here”:http://www.akkadia.org/drepper/cpumemory.pdf )</p>
<ul>
<li>
<p>in the eighties memory access was as fast as register access; (hi Sinclair and TI99/4A). In the nineties the CPU got much faster than main memory, on CPU caches are supposed to help, and must be understood in the wider context of hardware architecture.</p>
</li>
<li>
<p>Article focus on OS: linux + commodity hardware (Intel x86); standard Hardware architecture (as of 2007):</p>
<ul>
<li>CPU connect via Front side bus (FSB) to Northpbridge
<ul>
<li>Data between two CPU’s and CPU to main RAM access goes through Northbridge;</li>
<li>DMA (copy from device to memory without CPU) goes through Northbridge</li>
<li>the Northbridge is point of contention.</li>
<li>Connection from Northbridge to RAM:
<ul>
<li>Norm (2007) DDR2 has two buses(aka channels); access is interleaved (!); Memory controller is part of Northbridge</li>
<li>High end: four memory controllers are connected to northbridge each one connecting to its portion of the RAM via its own bus (increases bandwith – multiple memory buses.)</li>
</ul>
</li>
<li>Northbridge communicates with Southbridge (which communicates with peripherials via specialized buses such as PCI/PCIE, SATA, etc)</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>RAM chips</p>
<ul>
<li>SRAM (S=static) is faster than (and more expensive than) dynamic DRAM (D=dynamic); that’s why a machine has both of them.
<ul>
<li>SRAM: 4 transistors to do one BIT of memory; does not need refresh cycles to renew content (keeps information while power is on) ; (SRAM is used for fast cache memory within processor or device)</li>
<li>DRAM:
<ul>
<li>information is kept by capacitator, so need refresh cycles to renew content. One transistor to access (access line raises signal and the Data line gets the output: has bit or no bit); When read the output of data line is also used to recharge the capacitator (clever).</li>
<li>Refresh cycle required – ones per 64 milliseconds (half of the time is refresh cycle (!)); can’t access memory during refresh.</li>
<li>Memory cells are on a 2-d matrix – fewer access lines are needed this way; downside: additional limit of access: one row and column line must be needed to access a bit – because of this it is easier to read a whole row of the matrix rather than addressing individual bits (!!). Row size is 32 or 64 bytes.</li>
<li>Long on the gory details of protocol to access a DRAM bit is in Section 2.2 (didn’t sit this one out); throughput: 10GB/second on DDR2 (two channels); and 40GB/second on FB-DRAM (six channels). Speed didn’t change much over time due to physics.</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>CPU Caches</p>
<ul>
<li>fast SRAM is used for on CPU cache; uses the fact that we access the same data over and over again (loops in code – spatial locality; a program also works on related/same data structures – temporal locality)</li>
<li>Author says: CPU cache size is (typically) Main memory size / 1000; still need to keep program’s data access limited and local – because there is parallelism, and we all want to fit in.</li>
<li>Kinds of cache:
<ul>
<li>code cache: holds decoded instructions (decoding takes some time on intel cpu’s)</li>
<li>three cache layers today: CPU talks to small L1I (layer one instruction) and L1D (layer one data caches) - (access costs around 3 CPU cycles); if no hit then go to bigger L2 cache (access costs around 14 CPU cycles); if no hit then go to L3 cache; if no hit then go to main memory via Front side bus (access is around 240 CPU cycles)…</li>
<li>accessing the cache: multiple core (each core a logical processor); a core has up to two hyperthreads (if one hyperthread stalls, the other tries to be run run – so a hyperthread has its register set (for almost all registers); most frequent cause for switch between hyperthreads is delay due to memory access)</li>
<li>Intel: each core on the same CPU has its own L1d cache (hyperthreads share the same L1d cache); but L2 cache is shared between cores (for quad core CPU one L2 cache is shared by two cores); shared L3 core. AMD: only L3 is shared. L2 caches are separate. (Lots of possibilities here)</li>
</ul>
</li>
<li>Cache organization
<ul>
<li>each entry is called ‘Cache line’ – run of 32 (nowadays 64) bytes of memory; each cache entry is identified by its ‘Tag’ (conveniently this addresses a single row of DRAM cells (!!)</li>
<li>When reading in cache line: must make room in L1d – evict existing line from L1d to L2 (in Intel each L1 line is in L2 – inclusive cache; in AMD not – exclusive cache). Eviction from L3 means copy to RAM (expensive).</li>
<li>the tag: 64 address: - 6 lower bits are masked away (offset within tag); remaining bits are split: lower part is the cache set (cache is further subdivided); higher bits are the tag</li>
</ul>
</li>
<li>cache designs:
<ul>
<li>fully associative cache: cache set is empty – each cache line can hold arbitrary tag; good for very small caches with several dozen entries – for indexing HW needs full comparator for each cache line (lots of transistors)</li>
<li>direct associative cache: the cache set is index into array of equal sized small caches; the tag value is treated as the offset into a particular cache area. Problem: heavily used cache set indexes will see a lot of swap-in swap-out activity.</li>
<li>Set associative cache: again cache set is index into array of equal sized small caches; the tag lookup within a small cache is like with fully associated cache. Best option: address collision will not result in swap-out of heavily used cache line. N-way set associative cache := each small cache is N bytes large (N is a power of two). (says that larger than 8-way offers little improvement – however with more cores it will need still more N values – due to cache collisions between hardware threads).</li>
</ul>
</li>
<li>Evaluation (lots of interesting tests that will probably look totally different on slightly different hardware), tests check different access patterns (linear, random) + different working set sizes; see how memory access times differ – jumps respectively in access time when the relevant cache size is exceeded.</li>
<li>Writing:
<ul>
<li>Write-through cache each modified cache-line is immediately written to RAM (inefficient – each modification of a local variable is very slow)</li>
<li>write-back cache each modified cache line marked as modified(dirty); when it is removed from cache it is written to RAM; problem: syncing cache of two processors (cache coherence – both CPU’s have same picture of RAM).</li>
<li>Special modes (only kernel code can do this – sets MTTR register for a given address range)
<ul>
<li>uncacheable – acts like a write-through cache-line</li>
<li>write-combining – only goes to write when flushed explicitly (for memory mapped devices).</li>
</ul>
</li>
</ul>
</li>
<li>
<ul>
<li>Cache coherence: MESI protocol – the CPU’s synch each other on the status of cache lines.
<ul>
<li>Modified – cache line status: modified; no other processor has cached this line. (CPU that modified it had to send RFO messages to other cpu’s, so that they mark this cache line as Invalid)</li>
<li>Exclusive – cache line status: not modified; know that no other cache has this line. (no need to send RFO when modifying it)</li>
<li>Shared – cache line status: not modified; know that other caches can have it. (CPU1 hash sent Modified cache line to other processors when he wants to read the cache line and to memory controller)</li>
<li>Invalid - cache line not used. (CPU1 has sent modified cache line to other processors CPU2 (CPU2 has sent RFO (Request for ownership) message to CPU1) when he wants to write to the cache line)</li>
</ul>
</li>
<li>Summary: need RFO messages when (1) data is shared between threads on different cores (2) thread migrates to another core. RFO messages are slow as sender has to wait till they have been delivered via FSB (as a prerequisite for transiting to another stage)</li>
<li>One thread reads the other thread can modify: this doesn’t scale beyond two threads; four threads as fast as two – all due to RFO syncing !</li>
</ul>
</li>
<li>Which address is used as tag value? In L1 lookup they use the virtual address (? how, it is ambiguous, in two processes it can mean different things ?) because it is faster (address translation takes time); L2,L3 lookup is done with physical address (after address translation); Even more complicated: with HW virtualization the VMM is doing virtual address translation – not the processor (???)</li>
<li>Cache replacement strategy: Least recently used cache line is out (LRU) (? how is LRU list maintained ?);</li>
<li>instruction cache: L1i holds decoded instruction (decoding stage takes time); L2,L3 holds non decoded instructions; does branch prediction and pre-fetching (of both possible branches?); also instruction cache does not use MESI protocol (but simplified version) – it assumes that code is not self modifying</li>
</ul>
<p>Virtual memory</p>
<ul>
<li>Address translation
<ul>
<li>Address translation is done by HW (MMU – memory management unit in x86); but OS has to fill in the page tables. Page table forms a tree; highest order node is an array; each entry points to next level (which is also an array); etc. (each process has its own page table tree)</li>
<li>Virtual address is divided into parts [Level N index][Level N-1 Index]…[page offset].</li>
<li>Lookup: first the root node of page table is looked up (Level N index); this points to next level where [Level N-1 Index] is used to look up the next level. At the end you have the phys. memory page: here [offset] is used to find the physical address.</li>
<li>CPU: caches the result of page table walking (locality of reference says that most addresses are accessed within a few virt. Memory pages); the cache: TLB – translation look-aside buffer. So there is a L1TLB, L2TLB …) For L1TLB there is one for instruction L1ITLB and one for data L1DTLB)</li>
<li>On context switch the old TLB cache is no longer relevant and must be flushed (therefore each cache entry tag also includes the tree root as part; to make flushing easier) Flushing is done on exiting kernel space (!even though it could be used on return from kernel – if it did not switch contexts) (context switch is done by kernel); (Also virtual machine monitor that don’t share address space with host OS need to do that);</li>
<li>TLB cache is small because of frequent flushing.</li>
<li>Optimizations: if TLB tag has root of page table then CPU can invalidate only part of the cache that refers to given page table root.</li>
<li>Optimization: fewer page walks if pages are larger (problem: wasted physical space if page is not full). X86 has option of 2MB pages.</li>
<li>Virtualization: VMM (monitor) and guest OS have separate page tables (this protects against rogue guest OS). This makes for another TLB cache flush every time that control passes between kernel of guest OS and VMM. Now there are tricks to avoid this
<ul>
<li>processors virtualization extensions:</li>
<li>KVM (a VMM project) KVM is a linux kernel that is running guest OS (also linux); but the guest OS is not running in separate ring/virtual machine – it is running along VMM,</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>NUMA (non uniform memory architecture)</p>
<ul>
<li>Problem multiple processors in the same machine access the same memory – all memory access is via Northbridge, so we must manage how this is done. The trick is that CPU’s have their own local memory (access not via Northbridge).
<ul>
<li>Hypercubes: all CPU’s form an n-dimensional cube; one CPU (one node) is connected to northbridge. (can have 2,4,8 processors – more dimensions of the cube means more hops until it gets to node connected to northbridge</li>
<li>costs of access (estimate based on AMD machine) same node: read-100%, write-115%; one hop: read- 105%, write-130%; two hops: read-130%, write-150%</li>
</ul>
</li>
<li>OS support: should use local memory for processes code and data section (minimize use of Northbridge to get to RAM);</li>
<li>info that reveals NUM configuration in linux: access /sys file system (like /proc but more organized)
<ul>
<li>/sys/devices/system/cpu/cpu[n]/cache – n is #of core. Describes core/processor caches (L1,L2) shared_cpu_map column is bitmap – (each core #n gets its – the nth bit set) tells if the cache is shared with another core (means two cores are part of the same processor). (libNUMA accesses all this /sys and /proc goodness as a API)</li>
<li>/sys/devices/system/cpu/cpu[n]/topology – says which processors the cores belong to?: core_siblings – bitmap OR - which cores are on the same CPU. physical_package_id – which # CPU does this core belong to.</li>
<li>/sys/devices/system/node[n] – more about NUMA architecture (n is #processor) distance – array of values, distances in hops to other CPU’s (in hypercube). Distance=1 – means local.</li>
<li>/proc/<pid>/numa_maps – for shared memory regions – what is its status? (dirty-modified or not) and on what numa mode does memory for it come from?</pid></li>
</ul>
</li>
</ul>
<p>What programmers can do/optimizing memory access patterns</p>
<p>*setting memory only – problem that cache line is brought in, modified – stays in cache until flushed out; now it uses up valuable cache; better to bring in cache line, modify it and swap it out immediately - with sfence barrier instruction. (also called non temporal write) ; writes must be aligned to 16 bytes – all multimedia extension stuff needs this alignment. Example to set a cache line to zero (used to set big array in row wise access ( no improvement - write combining; CPU is good at optimizing sequential access by prefetching !) and column wise access (some improvement; but column wise access is three time slower by itself !)</p>
<blockquote>
<pre>
<code>
#include <emmintrin.h>
void setbytes(char *p, int c)
{
__m128i i = _mm_set_epi8(c, c, c, c, c, c, c, c, c, c, c, c, c, c, c, c);
_mm_stream_si128((__m128i *)&p[0], i); _mm_stream_si128((__m128i *)&p[16], i);
_mm_stream_si128((__m128i *)&p[32], i); _mm_stream_si128((__m128i *)&p[48], i);
}
</code>
</pre>
</blockquote>
<ul>
<li>Optimizing access patterns: matrix multiplication; transpose second matrix before multiplication to optimize mem. Access (to avoid column wise iteration) – this cuts time to 23.4% - five times faster !)</li>
<li>Outer loop iterates over cache line; inner loop multiplies sub matrix within a cache line (more complicated code) – cuts time to 17.3%</li>
<li>unrolling inner loops of multiplying cache-line sized sub matrix + prefetching by means of magical instruction – down to 9.47 !</li>
<li>trick: passing level 1 cache size to program: gcc -DCLS=$(getconf LEVEL1_DCACHE_LINESIZE)</li>
<li>L1 Data structure accesses
<ul>
<li>structure layout is displayed by pahole ; can optimize by minimizing holes between structure members (pahole –reorganize : recommends what to do)</li>
<li>layout of struct: put members that is accessed first as first struct member; access struct members in order that they are defined; If a structure spans two cache lines (misaligned) then it takes more cache line loads/stores to work with it – that makes misaligned access between 40%-400% slower (depending on size of working set – difference is smaller for large working sets)</li>
<li>structure elements used together should be close together – this minimizes conflict misses (the same cache set swaps out lines due to heavy usage) (cache set index is encoded in lower bits of the address)</li>
<li>Another trick: group data that is frequently used together into the same struct – stuff that is not frequently used can be put into a different structure (more effective prefetching on common use cases)</li>
<li>stack alignment – compiler tries to align stack frame to some known value (default 16 on gcc with x86) (but alloca messes things up). Stack must be initially aligned</li>
<li>Alignment: can force struct to be aligned to cache line size (careful – may take up more memory)
<ul>
<li>posix_memalign – memory allocation with requested alignment (but will consume more memory).</li>
<li>__attribute((aligned(64)) for static data</li>
</ul>
</li>
<li>L1 Instruction cache
<ul>
<li>processors do branch prediction – predict the address of jumps so that they can prefetch the code at target of jump. Moving branches out of inner loops is a good idea.</li>
<li>Code size should be kept small – inlining and loop unrolling may not give much (and make things worse!) gcc -Os : optimize for size ; it does do less inlining but attribute(always_inline) would force it to inline in any case (OMG, what about C++ inline ?).</li>
<li>Give hint to CPU about which branch is more likely: __builtin_expect (often wrapped by macro likely, unlikely) ; hash an effect if -freorder-blocks is set (is off in -Os but is on by -02) ; -freorder block will move the frequently used code right after the conditional jump, so that prefetching is likely to pick it up. The infrequent choice will jump further away.</li>
<li>Code alignment. -falign-functions=N : start of function is aligned to > 2 ^ N bytes ; to make prefetching the start of a function easier. -falign-jumps=N : same for start of basic blocks that are only reached by jump; -falign-loops=N : for start of loops (may be wasteful)</li>
</ul>
</li>
<li>L2,L3 caches; consider: L2 caches are shared between cores/hyperthreads; so a program can make use only of a subset; should match the working set size to cache set (like matrix multiplication should subdivide into small matrix so that it fit the cache). Compute available cache size: /sys/devicces/system/cpu[n]/cache – find the L2 size cache and subdivide it by n of bits in shared_cpu_map</li>
<li>prefetching of data :
<ul>
<li>hardware prefetching : is done when CPU detects an access pattern (two cache hits of a sequence of pages (or pages at the same gap, or linear access pattern (!)) – then CPU decides to prefetch the third one; each cache (L1d,L1i,L2) can have a special circuit that detects patterns and decides on prefetching (!). Prefetching limited to the same virtual memory page (don’t want to trigger swapping). Can stop Hw prefetch by emitting ud2 instruction (hints that we will jump to another location shortly)</li>
<li>software prefetching – not limited to the same virtual memory page ; must call function to explicitly load a cache line (helps a bit (8-10% with large working set sizes – larger than cache; can make matters worse, must be tested; different machines will have differently – behavior : hard to do). gcc -fprefetch-loop-arrays : does emit prefetch on loop iteration</li>
</ul>
</li>
</ul>
</li>
</ul>
<blockquote>
<pre>
<code>
#include <xmmintrin.h>
enum _mm_hint
{
_MM_HINT_T0 = 3, /*put into all caches*/
_MM_HINT_T1 = 2, /* put into L2 */
_MM_HINT_T2 = 1, /* put into L3 */
_MM_HINT_NTA = 0 // non temporal write; on eviction write to RAM.
};
void _mm_prefetch(void *p, enum _mm_hint h); // p – ptr; invalid values are ignored (alignment?)
</code>
</pre>
</blockquote>
<ul>
<li>OOO – out of order execution : the processor reorders instructions – it still has to evaluate instructions that produce the output for the next instruction, but there are several queues of dependencies, so it can switch to evaluate an instruction from a different/non conflicting queue if it would have otherwise stalled to wait for an additional operand be read from RAM. It gets complicated: the cpu tries to predict things – speculative loads …. (too complicated).</li>
<li>Prefetching by thread: can schedule special thread on the same core (but different hyperthread) (need to set affinity for that – libNUMA can help here !) – it does prefetching of the next 100 list nodes; speed up between 10-50% (depending on working set).</li>
<li>Device DMA (network device copies incoming packet into RAM – without CPU. Problem: CPU has to parse packet header this results in cache miss. The technique to fix that: devices pushes to cache of CPU that is about to be notified of packet arrival : DMA transfer has DCA flag; CPU sees this flag raised on Northbridge and puts header data into lowest level CPU cache.</li>
<li>Multi-threading
<ul>
<li>Concurrency problems: false sharing – the same cache line has two variables; two threads are writing each one to its variable; both thread caches need exclusive access to the cache line – a lot of RFO messages are send. Access time goes slower with more threads (on different cores). How to avoid false sharing for global variables?
<ul>
<li>Put shared read only variables into read only data section, make sure it is aligned . (declaring const)</li>
<li>thread specific variables into TLS (__thread specifier)</li>
<li>move data accessed together into its own struct; if there is contention between threads then make this struct aligned to a cache line __attribute((alined( CACHE_LINE_SIZE )) - (cache line size is preprocessor define set in cmd line)</li>
</ul>
</li>
<li>atomic access; not result is guaranteed if two threads access the same memory – atomic operations do (always sync with RAM – so it takes at least 200 CPU cycles); these can be subdivided further,
<ul>
<li>compare and swap (CAS) ; most common atomic operation - does the following atomically:</li>
</ul>
</li>
</ul>
</li>
</ul>
<blockquote>
<pre>
<code>
template<type T> bool CAS(T newValue,T *locationPtr, T expectedValue) {
if (*locationPtr == expectedValue) { *locationPtr = newValue; return true; }
return false;
}
</code>
</pre>
</blockquote>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> * load lock/load store (LL/SC) load lock operation starts transaction; load store – store memory on condition that load lock returned true. (have either CAS or LL/SC – CAS is more common)
* atomic arithmetic (use these if available instead of CAS – its faster)
* bit test – modify a bit of the value and return if it was set before.
* Memory bandwidth : more cores that can address memory you will use the full memory bandwidth to full potential (FSB bus speed is the limiting factor). What can be done?
* Identify problem: CPU counter NUS_BNR_DRV – counts cycles that core was waiting for FSB bus to get ready.
* Prevent threads from moving between cores (such migration incurs unneeded memory copy) – thread should stay on same core (set thread affinity – pthread_setaffinity, pthread_getaffinity) ; threads working on same data set should work on cores of the same processor (if cores on same CPU have shared L2 or L3 cache they can profit from this); threads working on disjunct data sets should not.
* NUMA considerations:
</code></pre></div></div>
<p>Tools</p>
<ul>
<li>diagnostics, measuring, profiling
<ul>
<li>oprofile records CPU counters (measures counters once per time period; counter value not always 100% accurate ) – problem: hard to interpret these correctly (and different CPU models have different counters) ; here you must look at multiple counters and their dynamics to get a sense of what is going on : L1D_REPL (l1d misses per instruction) ; DTLB_MISSES, L2_LINES_IN (consult Appendix B – or CPU manual)</li>
<li>Page fault counter – maintained by OS : Major page fault – needs to read disk to get virtual memory page; Minor page fault – caused with anonymous shared memory mapping that was not backed by file. time(1) reports these numbers; getrusage for the same process; wait4 – for a stopped child process. Also see /proc/<pid>/stat</pid></li>
<li>learning more about CPU cache : valgrind –tool=cachegrind <cmd line=""> (needs debug info): simulates the CPU cache, counts cache usage per function; advantage: more accurate than periodic measurements; can simulate size of L2 cache and n-way. disadvantage: it does not simulate context switches, so cache flushes are not considered = actual numbers of cache misses will be larger. Also valgrind introduces order of magnitude slow down...</cmd></li>
</ul>
</li>
<li>profiling memory usage: valgrind –tool=massif <cmd line=""> ; records memory allocations and stack us age - per function over time; memusage – displays total of heap size and (optional) sum of sizes of shared memory mappings ; displays dynamics over time</cmd></li>
<li>Branch prediction – can do better than __builtin_expect . PGO – program guided optimization
<ul>
<li>build the executable with profiling info (gcc: add -fprofile-generate - similar to coverage analysis – build creates .gcno files);</li>
<li>run a few tests – during runtime it records which choice for if .. else … is more common (at program exit if writes .gcda file for each object file) ;</li>
<li>Build the program again (with -fprofile-use ) – this time apply the gathered information to produce code with correct branch hint.</li>
</ul>
</li>
<li>Optimizing page faults
<ul>
<li>measure: valgrind –pagein : records page fault as they happen.</li>
<li>possible to reorder functions so that those used together are adjacent – reduces load time by up to 5% (tools?)</li>
<li>mmap can tell the OS to do a page fault on all pages (very expensive); but can be useful if the memory will be used soon (???) posix_madvise with flag POSIX_MADV_WILNEED can bring in all pages within a virtual address range (more fine grained).</li>
<li>Larger pages – less page faults (but more unused physical memory).
<ul>
<li>(architecture specific) On Intel the page size is 4k ; there are larger page sizes available and it is a linker option that affects the output binary to choose another format (?)</li>
<li>Huge page sizes: 2MB page size is available – but this lead to very high fragmentation; set of huge pages is manged as OS resource ; during boot (when continuous physical memory is still available) one can reserve 5 huge memory pages by echo “5” > /proc/sys/vm/nr_hugepages (needs root access).
<ul>
<li>
<table>
<tbody>
<tr>
<td>To get a huge page : shmget( “aa”, HUGE_TABLE_ENTRY_SIZE * n, SHM_HUGETBL</td>
<td>SHM_R</td>
<td>SHM_W</td>
<td>IPC_CREAT); // size must be multiple of huge table page; fkag SHM_HUGETBL indicates to use a huge page</td>
</tr>
</tbody>
</table>
</li>
<li>each huge page is represented as a pseudofile in the hugetabl file system (/proc pseudo file system); So we can mount this file system and then get the file name of an available huge page and map it to memory with mmap.</li>
<li>Test program with huge table was 57 faster – illustrates that a large cost is TLB misses + VM paging.</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>The Lost Art of C Structure Packing by Eric S Raymond ( “here”:http://www.catb.org/esr/structure-packing/ )</p>
<p>I wrote a short C program that sums up the article; the output of the program on a 32 bit processor is as follows (program is “here”:https://github.com/MoserMichael/cstuff/blob/master/training/align.c)</p>
<blockquote>
<pre>
<code>
! Purpose: #pragma(pack) produces small structures, but slow code to access them! (interrupt to handle misaligned access)
! Purpose: want to use unaligned structs in a space efficient way
! a structure field of uint8_t can be on any address
! a structure field of uint16_t must be on address divisible by two.
! a structure field of uint32_t must be on address divisible by four. (etc)
sizeof({ uint16_t, uint8_t })=4 (0,2) sizeof({ uint8_t, uint16_t })=4 (0,2)
! to minimize space: try to pack more than one structure field into the holes between the structure fields
sizeof({ char8_t , uint16_t, void* })=8 (0,2,4) sizeof({void *, uint16_t, uint8_t })=8 (0,4,6) sizeof({uint8, void *, uint16 })=12 (0,4,8)
! an array of uint8_t can be on any address (same as uint8_t field)
! an array of uint16_t must start on an address divisibe by two
sizeof({uint8[sizeof(void*)+1], uint8_t})=6 (0,5) sizeof({uint8_t, uint8_t[sizeof(void*)+1]})=6 (0,1)
sizeof({uint16_t[3], uint8_t})=8 (0,6) sizeof({uint8_t,uint16_t[3]})=8 (0,2)
! nested structures are a source of some waste of memory due to padding/alignment (hi c++)
! One way is to reorder the structure to fit into a smaller size
sizeof({uint8_t, { void *, uint16_t }})=12 sizeof({void *, {uint8_t, uint16_t }})=8
! bitfields: a series of bitfields is aligned to word size (sizeof(int)=4)
! because bitwise instructions are used to implement them (these work on ints)
! common convention to have a padding field that pads up to word size
! also: always use unsigned type for bitfield values
! still: they try to pack them as close as possible (!)
sizeof({ uint_t a:3; uint_t b:4; uint8_t c:2; }) = 2 (!!!)
sizeof({ uint16_t a:15; uint8_t b:2; })=4; sizeof(int)=4
! bitfield member that does not fit the current word is moved to the next word
sizeof({ int a:31; uint8_t b:2; })=8
! When in doubt obtain the layout of a struct by means of offsetof (can't take offsetof bitfield members)
! or use clang's -Wpadding or use pahole.
! double or long double fields can have surprising behavior
! thank you !
</pre>
</code>
</blockquote>
Now it seems that unaligned memory access on new Intel processors (Sandy bridge and newer) is as fast as access that is aligned. ("here":http://lemire.me/blog/2012/05/31/data-alignment-for-speed-myth-or-reality/ ) It is different if you structure spans multiple cache lines - but as long as you stay within the same cache line it is the same.
Compilers will not change all structures to packed - after all on other platforms such as ARM/RISC you still can't work with unaligned access, but if the program works on new Intel processors only, then you are fine with packed structures.
In fact packed structures may help - more data fits into one cache line, so that fewer cache lines are accessed.
Interesting how old assumptions do have a habbit of turnout out wrong ...
More to follow ...
</pre></blockquote>
David Graeber on Bureaucracy
2015-03-23T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2015/03/23/bureaucracy
["", "ramblings"] | <h2 id="david-graeber-on-bureaucracy">David Graeber on Bureaucracy</h2>
<p class="publish_date">
23 March 2015
</p>
<p>‘The Utopia of rules’ by anthropologist David Graber is an interesting study of modern bureaucracy and where it came from. (Can be purchased <a href="http://www.mhpbooks.com/books/the-utopia-of-rules/">here</a> )</p>
<p>The author argues that since the seventies there has been a subtle shift away from social democratic corporatism (the idea that management and workers have shared interests) towards the practice of unrestrained value extraction by management. This shift comes together with a strengthening of corporate bureaucracy and the weakening of shareholders;
Now most new (bureaucratic) regulations have the hidden aim of extracting even more fees from the subjects, so that it all serves the grand scheme of unrestrained money grabbing.</p>
<p>later bureaucratic forms of management were pushed into every aspect of existence, as the corporate and state bureaucracies merged into a single whole.</p>
<p>The resulting picture is rather bleak; the author even also compares the resulting bureaucratization as being similar to that of the former Soviet Union.</p>
<p>(i think that there is an alternative explanation for a marked rise in the amount of bureaucratic procedures since the seventies: nowadays everything is looked at as a <a href="http://en.wikipedia.org/wiki/System">system</a> ; this makes the analysis and resulting management structures much more complicated, now with a more complicated management structure we now have a more complicated system of bureaucratic procedures. If all you have is a hammer then everything looks like a nail - system science and computers would be this universal hammer)</p>
<p>However one aspect is left out of this picture: the degree of personal choice permitted/tolerated by this system of governance is still much larger then anything that existed in the Soviet Union; one possibility is that people will hack their way out.</p>
<p>I would define the practice of ‘hacking’ as follows: a hacker analyzes the structure of a system, each part of the system may have conflicting requirements: for example what is permitted on a conceptually lower level may not be allowed on a higher level (and vice versa); however the different parts of the system do interact in order to constitute the system ‘as a whole’. Of course these discrepancies can be exploited with positive results; in practical terms that would mean that a better understanding/command of the bureaucratic system can lead to a more efficient ‘march through the institutions’.</p>
<p>If this hacker mindset reaches a wider audience then this will allow for customization/gradual change/reformation of the bureaucratic system of governance.
Or maybe it will just result in a backlash that will turn the system into something even more authoritarian; who knows.</p>
<p>I still think that gradual change is much more productive than attempts of a universal overhaul/revolution of cosmic scale.</p>
<hr />
<p>my summary of ‘the Utopia of rules’ by David Graber</p>
<hr />
<p>Essay ‘The Iron Law of Liberalism and the Era of Total Bureaucratization’</p>
<p>-sixties and early seventies: rebellion against bureaucracy
-later period: less public interest in the topic; but more and more paperwork is required !
paperwork became ubiquitous, so we don’t notice it; how did this change come about?</p>
<p>The right had the liberal critique of bureaucracy := free market mechanism are more efficient preferable to government intervention/bureaucratic management. 19th century liberals saw bureaucracy as a remnant of the Feudal order; Ludwig van Mises - bureaucracy is a flaw of the democratic process - information less efficiently organized than by free market; bureaucratic rule leads to dictatorship.
(Q: the author did not mention Hayek who said similar things)</p>
<p>Problem: number of civil servant grows under all political orders (the ‘iron law of liberalism’ is that action to remove red tape for the sake of liberalism produces even more red tape)
(Q: not limited to ‘liberalism’, socialists are also very successful at generating red tape, maybe even more than liberals ?)</p>
<p>Markets were initially created as side effects of military operations (Rome: procurement for the army, coinage in Rome invented as means of provisioning); later: central banks invented to finance war.
(Q: in Babylon it was the temple/palace economy with the aim to organize long distance trade for raw materials)</p>
<p>Also they need a lot of regulations to make a free market happen (it doesn’t happen by itself) “an endlessly ballooning array of legal clerks, registrars, inspectors, notaries, and police officials who made the liberal dream of a world of free contract between autonomous individuals possible. It turned out that maintaining a free market economy required a thousand times more paperwork than a
Louis XIV-style absolutist monarchy.”</p>
<p>“The rise of the modern corporation, in the late nineteenth century, was largely seen at the time as a matter of applying modern, bureaucratic techniques to the private sector” impersonal bureaucracy this was supposed to be better than ‘the networks of personal or informal connections that had dominated a world of small family firms’ (Max Weber wrote a lot on the impersonality and objectivity of the bureaucracy)</p>
<p>The USA became very bureaucratic when it turned into a world power (early twentieth century); the state adopted the bureaucratic machinery of the big private corporations; (so both bureaucracies kind of fused - in both public perception and practice)</p>
<p>This private/public fusion was greatly increased by the New Deal (practice like ‘revolving doors’ where civil servants turn into industry managers and vice versa) and was even more fueled by the need to administrate the WW II wartime economy; the author says the US never demobilized, so it essentially stays like this ever since.</p>
<p>“the need to preserve certain domestic industries for military purposes, and to develop others, has allowed the U.S. government to engage in practically Soviet-style industrial planning without ever having to admit it’s doing so. After all, pretty much anything, from maintaining a certain number of steel plants, to doing the initial research to set up the Internet, can be justified on grounds of military preparedness. Yet again, since this kind of planning operates via an alliance between military bureaucrats and corporate bureaucrats, it’s never perceived as something bureaucratic at all”</p>
<p>Growth of importance of the financial sector (after scrapping of the gold standard in 1971 / post Bretton woods) created even further ‘regulation’ - as the state has to regulate the amount of money creation and keep up some semblance of competition. Fusion of private and public sector means that these regulations are greatly influenced by the Banks.</p>
<ul>
<li>
<p>before 1971: dominant theme was corporatism: long term employment in mega corporation created impression of shared interest between management and workers (both white and blue collar); investors were subordinate to this corporatism.
(sixties radicals were up against this coalition so they were talking about social-fascism like the commies in Weimar of the twenties)</p>
</li>
<li>
<p>after 1971: new focus of management on value extraction (away from corporatism) + tended towards shareholders and finance. Now no more long term employment, if management wants to win loyalty it grants shares - circle of investor class was widened. So that the middle classes bought the idea that they have a stake in the investors fortune (this would isolate them against the unions/blue collar workers)</p>
</li>
</ul>
<p>Propaganda: everyone should look at the world through the eyes of the investors, CEO’s are the new heroes of the nineties. Now investors/management goals and middle class goals were aligned with middle class - this ‘set the stage for the process whereby the bureaucratic techniques (performance reviews, focus groups, time allocation surveys ?) developed in financial and corporate circles came to invade the rest of society-education, science, government-and eventually, to pervade almost every aspect of everyday life’; Corporatism like ‘vision,best practices’ invaded language
(Q: what about the left with PC, didn’t they help with adoption of newspeak ?)</p>
<p>‘For all its celebration of markets and individual initiative, this alliance of government and finance often produces results that bear a striking resemblance to the worst excesses of bureaucratization in the former Soviet Union or former colonial backwaters of the Global South’</p>
<p>Sign of this: the cult of diplomas and certificates - these were very important in bureaucratic societies of SU and former colonies and seen as a symbols that impart authority; now in US you need a BA for sort of everything. In the US student loans put them students into long term debt, new formal education requirements mean even more debt (the union of public and private bureaucracies want to
extort a cut off from future income)</p>
<p>Says that this is likewise in finance: In US banks that are caught swindling by the regulator are not brought to court, instead they settle for a fine that is less than the profit gained by transgression!</p>
<p>Lots of built in double standards; why?
Bureaucracy is an Utopian project: claims to be based on meritocracy (formal examination as entry criteria), but being a cousin of a boss is also important.
so pretending that the Utopia is real leads to double standards:</p>
<p>Says that protests against globalization/free trade agreements in the nineties were protests of the left against predatory bureaucracy (?) (later he regrets that this was not the real focus)
claims that protest actions were meant to break the wall of censorship
‘Lessons learned’ from this movement:</p>
<ul>
<li>
<p>impersonal rules and regulations only effective if backed by threat of violence (that’s why 19th century liberalism saw introduction of modern police); More bureaucratic control -> more heavily armed police, strong threat of violence against transgressions; also nobody dares to resolve conflicts without calling the police.</p>
</li>
<li>
<p>technology is not the cause of processes it is a tool; the key was institutional change (alignment of finance and corporate bureaucrats) ‘the direction that technological change has taken is itself largely a function of the power of finance’</p>
</li>
<li>
<p>all these changes were justified by (bureaucratic) efficiency and (market rationality); when ultimately it is all about money extraction ‘talking about rational efficiency becomes a way of avoiding talking about what the efficiency is actually for’. Says that a left wing critique should show how all these changes fit together to extract/extort money.</p>
</li>
</ul>
<p>In the 19th century Americans used to think that value is produced by labor and that capital derives from labor (and is therefore secondary); this changed with the big corporations, these pushed a new ideology ‘Andrew Carnegie spoke of it as “The Gospel of Wealth” - that value was instead derived from capital itself’ and that value is ultimately produced by bureaucracies of big corporations.</p>
<hr />
<p>Public debate about bureaucracy is largely out of touch:</p>
<p>George Wallace in 1968 created the following narrative: bash them parasitic politicians and condescending bureaucrats! (this caught on because the US likes the 19th century picture/ideal of self sufficiency); this narrative was later adopted by both Reagan and Clinton; this line of argument was very successful because Americans like to see themselves as self sufficient individualists.</p>
<p>The term ‘deregulation’ means increasing more regulation to achieve the desired effect such as more competition in the financial sector. Now the left says this created the crisis of 2008 - so they are seen as being for even more regulation.</p>
<p>later the welfare state was scrapped; the left would just try to save remnants of the old welfare state, while trying to adapt ‘modernizations’ - some fusion with privatization that only caused Bureaucracy to grow;
-> the left can’t formulate any positive alternatives.
(the authors critique of bureaucracy is supposed to contribute to a discussion toward one)</p>
<hr />
<p>Essay ‘Dead zones of the imagination’</p>
<p>Why anthropologists did not study bureaucracy - but they do study ritual surroundings in society, and bureaucracy is part of the rituals (it goes with both, death, everything).</p>
<ul>
<li>
<p>paperwork is boring; in contrast to that rituals are dense and can be interpreted without end.</p>
</li>
<li>
<p>But literature does deal with bureaucracy? answer: ‘Great writers, then, know how to deal with a vacuum. They embrace it. They stare into the abyss until the abyss stares back into them. Social theory, in contrast, abhors a vacuum-or, this is certainly true if its approach to bureaucracy is anything to go on. Stupidity and violence are precisely the elements it is least inclined to talk
about’ (Q? violence is not interesting to social sciences ?)</p>
</li>
<li>
<p>social scientist did not produce a critique of bureaucracy - despite a growing amount of administrative procedures in academia.</p>
</li>
</ul>
<p>Maybe because they are building these structures? academia as a factory of workers for the US/global administrative apparatus.</p>
<p>Weber was very important in the sixties in US social science: Max Weber was adapted to act as developmental theory for the world bank (to counter Dialectical Materialism/Marxist ideology). (Also Weber was a big fan of bureaucracy : he sort of formulated its meritocratic ideals and claims to objectivity)
And up until Vietnam the social science departments cooperated with the military intelligence apparatus (Q: he does not mention that this deep cooperation goes back to WWII)</p>
<p>After Vietnam the social science departments turned leftwards; now Weber was out and Adorno/Frankfurt school was in; later Foucault became the new unquestioned authority (what Weber used to be
before)
(they still used Weber for ‘rational choice theory’ - in order to train new bureaucrats)</p>
<p>Foucault said that knowledge is a form of social power (the Power/Knowledge Nexus); social sciences like to interpret ‘dense’ topics (with the hope that their skills at interpreting these will place them in a position of power); Bureaucracy is not ‘dense’ so it is not handled. (Q? Maybe they just don’t want to confront their real masters ?)</p>
<hr />
<p>Back to the theme of violence: “institutions involved in the allocation of resources within a system of property rights regulated and guaranteed by governments in a system that ultimately rests on the threat of force” (as punishment for non compliance, this is also known as ‘structural violence’)
People instinctively like to ignore this implicit thread of force - because its not addressed to them - if they comply.</p>
<p>Anthropologists don’t like to interpret ‘structural violence’ in these terms; they avoid looking at question of implicit threat of state power; instead they look at secondary things like gender relations/sexism, rituals etc.</p>
<p>(What follows is typical for anthropologists - an example from the very strange culture of Madagascar; a strange analogy to Bokononism in ‘Cat’s cradle’; can only guess what is true and if there is an error of interpretation)</p>
<ul>
<li>
<p>the pre colonial Merina kingdom had slavery; so people were used to master-slave relations.</p>
</li>
<li>
<p>the French took over in 1895 and abolished slavery; now everybody assumed that they were all in equal status as slaves of the French.</p>
</li>
<li>
<p>as a result the culture of the Malagasy subjects changed: now every power relationship between adults in society became an objectionable thing
(Q? so they all became hippies, i doubt that, in traditionalist societies people usually have to obey existing customs; sort of ‘tyranny of the commons’ ?)</p>
</li>
<li>the paradox is that the French (and pre colonial) masters were never really present at the local level, so this is an abstract fear: the bureaucracy did never provide any services to the
population, (no police and no regulations !))</li>
<li>the state bureaucracy did have to maintain an air of authority: they did that by using French (which became the language reserved to commands !);
also education is revered - and you need some of it in order to became a bureaucrat: so the position of bureaucrats is based on strange rituals</li>
</ul>
<p>Political science on relationship between coercion and information;</p>
<ul>
<li>
<p>if the subordinates fear the master then they are not inclined to share information with the masters (but the subjects need to understand their masters !)
(more coercion leads to less information)</p>
</li>
<li>
<p>also masters do not need to understand the subjects if they can just apply force.
Masters can then grow totally out of touch with reality.</p>
</li>
</ul>
<p>Still masters continue to apply bureaucratic procedure as a substitute for communication with their subjects (example: passport system that started apartheid; policeman no longer had to communicate with the blacks - only had to look at the information in the internal passport)
Violence removes the need to communicate with subjects; so subjects say that procedures that regulate this orders are stupid.</p>
<ul>
<li>
<p>less extreme example: Men in paternalistic settings say that (a) women act illogically (b) but women have ‘intuition’ (analogy: peasants are stupid but have ‘peasant wisdom’)
These schema help men to avoid the need of communication with their women.
However dependent women need to employ more effort in order to understand their masters.</p>
</li>
<li>
<p>now for subordinates: ‘Imagination tends to bring with it sympathy, the result is that victims of structural violence tend to care about its beneficiaries far more than those beneficiaries care about them. This might well be, after the violence itself, the single most powerful force preserving such relations’</p>
</li>
</ul>
<p>Anthropologists again don’t like to analyze violence as a means of communicating a message and as having symbolic powers (they also romanticize violence - see it as sending a message to a larger audience as a symbolic act)</p>
<p>(Q: ? Strange: when I communicate with social workers on behalf of some poor guy then I always get the impression that social worker is not inclined to help; they are just bureaucrats guarding the state money from encroachment; Maybe Anthropology is also not about understanding other cultures rather than about how to handle these cultures in a colonial/post-colonial framework ?)</p>
<p>So the main function of the Police is about enforcing regulations by projecting the implicit threat of force; much less often they do deal with crime-fighting.
“Police are bureaucrats with weapons.” Most times that people get beaten up by error is by just talking back; Policemen feels that his ‘ability to define the situation’ is being challenged and reacts non proportionately/(childish because he is unable to correlate different points of reference)</p>
<hr />
<p>In theory: Bureaucracy is about abstraction: the rich realm of reality is classified into ready made schema and statistics (formulas as tools of classification).
Of course each step of abstraction throws away details that are not relevant to the abstract schema.</p>
<p>Now the Bureaucracy has to support existing power relationships - so the important detail/biases of which details are thrown away is determined by these power relations;
If the abstraction process is out of step with reality (power usually get to this position as explained earlier) then situations of stupidity are created - the abstract schema of regulations perpetuate the ideas of the rulers that are out of step with reality (structural blindness) - this discrepancy is perceived by the subjects as idiotic.</p>
<p>(this perception of idiocy was exploited as a tool by the neo-conservatives: they had to mobilize support for scrapping the welfare state - despite the fact that it was in the interest of the subjects)</p>
<p>“structural violence creates lopsided structures of the imagination. Those on the bottom of the heap have to spend a great deal of imaginative energy trying to understand the social dynamics that surround them-including having to imagine the perspectives of those on top-while the latter can wander about largely oblivious to much of what is going on around them.”</p>
<p>(Q: ? Maybe subjects are calling it ‘stupidity’ partly because they are not trained in abstract methods of thinking ?).
(Q: ? Maybe calling these skews stupid is just a defense mechanism of the subject, as a way of rationalizing it while still staying remaining the larger framework ?)</p>
<hr />
<p>Says that the left was always acting as a critique of Bureaucracy (all protest movements had this component); Yet they all generated their own bureaucratic procedures ! Why?</p>
<ul>
<li>
<p>Part of this is due to realism - to give it all an orderly appearance the left does the same - it creates its own bureaucracy.</p>
</li>
<li>realism comes when one has to accommodate/face the threat of force</li>
<li>when the revolutionary movement creates its own realities it has to interact with regulations of the old order - hence the rise of procedures.
(? naturally for a leftist he blames it on the old order ?)</li>
</ul>
<p>Also the left failed to produce a critique of bureaucracy/ existing power relations. why?</p>
<ul>
<li>
<p>He says the 19th century left did not see question of institutional violence as primary factor (? why does he ignore Bakunin ?)</p>
</li>
<li>Marx thought that mode of production was more important; stressed the imagination required for production process (humans design their artifacts before they make them)</li>
<li>Sen Simon said that artists should be the vanguard; only they were supposed to have the imagination to create a new social order.
<ul>
<li>so both thought that the great liberation would be a liberation of imagination. These were the origins of the 68 slogan ‘power to the imagination’.</li>
</ul>
</li>
</ul>
<p>(Q: ? what liberation, for who?: Marx, Sen Simon and Lenin did not have any problems with coercion, as long as they are the rulers themselves (see Marx on the ‘dictatorship of the proletariat’ - after the Paris commune; also he ignores that in all ‘successful’ revolutions the anarchist idealists were put to the wall by the more organized managers. ?)</p>
<p>Discrepancy in Marx: when talking about imagination in material production he talks about ‘design up front’; when talking about revolution (the earlier Marx) stressed the that socialist society can’ be designed up front. Author says this is due to different types of imagination.</p>
<ul>
<li>in material production the creative work is done exclusively by the elite/rulers (this separation leads to alienation of the workers)</li>
<li>in social production (childcare, housework) interpretative work mainly done by subordinates (see before); Here alienation of subjects is due to living in a structural oppressive structure.</li>
</ul>
<p>Argues that both types inequalities are shattering imagination; and that ‘alienation’ is exactly this shattering of the imagination.
Bureaucratic procedures are ‘stupid’ because they codify situations of structural inequality “such procedures come to partake of the very blindness and foolishness they seek to manage”</p>
<p>(Q? and in a kibbutz/commune there is still someone who has to do the boring work)</p>
<hr />
<p>Big question: how does one change society without creating a new violent bureaucracy?
Of course they try to change peoples outlook (cultural revolution):
Situationist: ‘creative acts of subversion’ should give people a temporary break from settings that turn them into passive spectators - this liberates people to regain their creativity.
Paradox: upheavals occur out of nowhere; says that the public when put into a liberating contexts acts radically then in a normal setting.
(that’s why the same French public could be sympathetic of the students in 68 and still a month later elect a right wing government)
Its the surrounding institutions and procedures that are defining the ‘horizon of the possible’</p>
<p>()</p>
<hr />
<p>essay ‘On flying cars and the declining rate of profit’</p>
<p>In the sixties and seventies there were certain assumptions about technological progress in the near future that did not take place; instead we got computers (tools for simulation)
Says that formerly the number of scientific papers no longer doubled within fifteen years; from 1970 onward this stopped; speed of travel also peaked in 1971.
the author feels cheated; this essay tries to explain why this happens.</p>
<p>(Q? did it occur to the author that the more a technical system becomes complicated; the more complex it is to change it ? (and once you have to embed a computer it becomes even harder);
a bit naive to assume that it is a linear progression; diminishing returns: in the sixties they had to put in the enormous expenses of the space race in order to continue with fast growth, that
was unsustainable; also there is a limit you can get out of fossil fuels.)</p>
<p>Thesis
in the seventies they switched to favor technologies required for labor and social control.</p>
<p>(lots of bullshit follows)</p>
<p>Says that AT&T slashed investment into research due to changes in taxation: before the seventies corporate taxes were high so AT&T preferred to put profits into research.
(Q? does not mention the breakup of the AT&T monopoly in the late seventies ?)
Says that lower corporate taxes also meant higher executive salaries and stock buybacks (instead of research and capital investment)</p>
<p>Public research went into arms race; so that it all had a military slant (drones instead of multipurpose robots). (? problem: tech is just not yet there ?)
Also claims that technologies for social control were deliberately favored.</p>
<p>Antithesis
Claims that rulers figured that tech progress creates problems (Toeffler) so that changed allocation of funding to less disruptive areas.
Still even in well financed areas there was slow progress.
Claims that in big science also suffers from hyper bureaucracy (application of corporate stye management let to big increase of administration effort) with disastrous effects on science.</p>
<p>Also a big increase of competition (and privatization of results/patents) means less sharing and open discussion among researchers; It all amounts to an atmosphere that rejects brilliant individuals
and favors colorless bureaucrats (‘green colorless ideas sleep furiously’ - yeah the sentence has a sense!)</p>
<p>In 19th century Britain they did not have big corporations; so it was all governed by individual entrepreneurs; and that led to a climate that favored the oddball tinkerer.</p>
<p>Big corporations ™ came about in Germany and the US; hence the rise of big state sponsored big science programs (still they had a place for brilliant individuals);</p>
<p>“The final victory over the Soviet Union did not really lead to the domination of “the market.” More than anything, it simply cemented the dominance of fundamentally conservative managerial elites
-corporate bureaucrats who use the pretext of short-term, competitive, bottom-line thinking to squelch anything likely to have revolutionary implications of any kind.”</p>
<p>(Q? what about ycombinator/’angel’ investors: successful entrepreneurs who are funding new ventures ? he failed to mention that.)</p>
<hr />
<p>Synthesis</p>
<p>“administrative imperatives have become not the means, but the end of technological development. “</p>
<p>Marx through that competition between capitalists inevitably leads to technological progress; now “the current form of capitalism, where much of the competition seems to take the form of internal
marketing within the bureaucratic structures of large semi-monopolistic enterprises, would presumably have come as a complete surprise to them”</p>
<p>(bla bla Utopia will lead us out of stagnation, bla bla)</p>
<p>=</p>
<p>essay ‘The Utopia of rules, or why we really love bureaucracy after all’</p>
<p>Max Weber: a bureaucracy claims the monopoly on information within a government (each government guards his knowledge - thus creating job security), this way it makes itself indispensable to
government, (and impossible to get rid of it; rulers and dynasties change but the civil service remains)
Appeal of bureaucracy: its objectivity: in its ideal form it is “simple, predictable, and-within certain parameters, at least-treat everyone more or less the same.”</p>
<p>Well in the 19th/early 20th century the state socialism based on bureaucratic rule had a strong appeal (Hayek also says so, but here is a better explanation as to why this came about)</p>
<hr />
<p>In both Germany and the US of the 19th century the postal system was admired for its efficiency.</p>
<ul>
<li>
<p>The post applied top down military principles to organization. Postal services were first the couriers of the military; later commercial function was added, even later in the 19th century the
wider public started to use it.</p>
</li>
<li>
<p>In mid 19th century more than half of state’s budget went into the postal system ! (In the US 70% of all federal workers were employed by the postal service )</p>
</li>
<li>
<p>In Germany: the post office was the inspiration for Weber’s admiration of bureaucracy. in 1867 Prussia acquired the right to run its postal service for the whole of the Germany holy empire; it
was used as a tool for unification and was therefore a showcase/prestige project for the unified state.</p>
</li>
<li>
<p>the same principles of organization were later used to govern the German states social services (Bismark emulated socialist social insurance + other services after his the Sozialistenverbot of
1978)</p>
</li>
<li>
<p>Lenin’s notion of state socialism was inspired by the German postal service! (‘State and Revolution’ p 52)</p>
</li>
<li>
<p>Anarchist (Kropotkin) were inspired by a different aspect: postal service between states was regulated by the international postal union - by method of negotiations based on consensus (not
dictated).</p>
</li>
<li>
<p>Lenin’s ideas of state socialism were also based on in (‘State and Revolution’)</p>
</li>
<li>
<p>now the same pattern happened with the internet!</p>
</li>
</ul>
<p>”
A new communications technology develops out of the military.</p>
<ol>
<li>It spreads rapidly, radically reshaping everyday life.</li>
<li>It develops a reputation for dazzling efficiency.</li>
<li>Since it operates on non-market principles, it is quickly seized on by radicals as the first stirrings of a future, non-capitalist economic system already developing within the shell of the old.</li>
<li>Despite this, it quickly becomes the medium, too, for government surveillance and the dissemination of endless new forms of advertising and unwanted paperwork.
“
<ul>
<li>bureaucracy appeals when it is part of a technological system that achieves impossible dreams.
Here tech stands for rationality, which is giving us great Powers.</li>
</ul>
</li>
</ol>
<hr />
<ul>
<li>
<p>former (Medieval perception, also classics) says that rationality has the function of taming our animal instincts.</p>
</li>
<li>
<p>David Hume: rationality is a tool of achieving our goals; so here it is outside of morality.</p>
</li>
</ul>
<p>now “The whole idea that one can make a strict division between means and ends, between facts and values, is a product of the bureaucratic mind-set, because bureaucracy is the first and only
social institution that treats the means of doing things as entirely separate from what it is that’s being done”
Bureaucrats are supposed to be rational where rationality is divorced from ethics;</p>
<ul>
<li>these two interpretation coexist somehow, so it is not clear what the word ‘rationality’ actually means (can mean contradictory things in different contexts)</li>
</ul>
<hr />
<p>It gets worse:</p>
<ul>
<li>
<p>notion of rationality was first advanced by Pythagorean: they noticed that many things can be described mathematically - leading up to motion of heavenly bodies (planets)</p>
</li>
<li>
<p>later in late Roman times this notion was universally adopted; now the trick is that it was assumed everything originated with G-d; so rationality was used as a tool of understanding the
transcendental; so rationality was a tool of the mystical (at that time); \</p>
</li>
<li>
<p>so in late Roman times they tried to picture transcendental (angelic) hierarchies of beings where everything is ordered by increased level of rationality; this looked quite similar to the Roman
bureaucracy !
This went on in medieval Europe, even though they did not have a real state with a bureaucracy to begin with !
(Q? and the Roman church did not have a bureaucracy of its own ? i don’t buy that)</p>
</li>
<li>
<p>says that the Renaissance brought no clear break with the past: now reason as is turns into something transcendent (or</p>
</li>
</ul>
<p>“The appeal to rationality in Descartes and his successors remains a fundamentally spiritual, even mystical, commitment, that the mathematical or math-like abstractions that are assumed to be the
essence of thought, are also the ordering principles that regulate nature-and this remained true whether they were identified with God, or seen as the ultimate proof of God’s nonexistence.”</p>
<ul>
<li>in this tradition notions of rationality (or bureaucratic rationality) never remain to just as a tool; they always turn into some notions of cosmic hierarchy (or ‘higher reason’)
(for example we are told that reason is what distinguishes Humans from animals - that’s bullshit but apparently ‘reason’ still bears this transcendent role)</li>
</ul>
<hr />
<p>When speaking about states:</p>
<ul>
<li>sovereignty - an area where there is a monopoly on the use of force</li>
<li>administration - (bureaucratic norms and procedures of government)</li>
<li>politics ( any competition/interaction between the political actors/centers of power)</li>
</ul>
<p>On the fringes of centers of cultures (Mesopotamia,Egypt, Rome) there were ‘Heroic’ barbarian societies, where politics was very important as a permanent contest between aristocrats.
(also they are the negation of everything that the cultured bureaucracies stood for)
(what follows is a long cultural criticism peace on which principles of bureaucracy are negated in fantasy books)
Fantasy books about these ‘heroic’ society have an appeal for this reason; says its a form of escapism and reassurance that you you’d rather want to live in the real world.</p>
<ul>
<li>And this reassurance is the actual reinforcement of the bureaucratic order.</li>
</ul>
<hr />
<p>What is a game? - it has a clear designation of players, is bound in time and is defined by rules. Real world situations are ambiguous, and people follow them similarity -> a game is a utopia of
rules. (‘Game’ is in contrast to ‘play’, which is not governed by rules)</p>
<p>Sovereignty: in Medieval Europe they had this question: is the monarch bound by the laws of the country or is he above the law?
Personal power thrives (and becomes feudal) if the rulers are not bound to regulations (can play by their owns rules);
Another area - grammar: grammar is descriptive in nature; but once the grammar book has been written people look to it as a prescription.</p>
<p>So there is a tension between rules/game/grammar vs anarchy/play/free usage ; without rules there is despotism/lack of common framework/absence of common language; too much rules however turn into
a straight jacket and kill creativity.</p>
<p>If (any) organization grows in size then it has to organize somehow; however anarchists try to build non dominating structures, so this is a inherent contradiction for them.</p>
<ul>
<li>
<p>Jo Freeman: in feminist groups larger than twenty informal cliques start to form; they start to control(/hide) information from the rest of the group and start to wield power.</p>
</li>
<li>now they can either formalize this state by turning these cliques into governing bodies (steering committees); this process creates a lot of rules (the positive effect is that this can create
mechanisms of accountability)</li>
<li>the other choice is to not grant these cliques special status; Problem: they continue to wield real power from the shadows/non formally.</li>
</ul>
<p>(Q:? I would add that they can avoid creation of accountability structures and continue with arbitrary rule - but the author likes the idea ? He dismisses the argument that groups that are ruled
by shadow structures as an ‘esthetic argument’)</p>
Roget Thesaurus parser
2015-02-26T00:00:00+00:00
http://mosermichael.github.com//all/projects/2015/02/26/roget
["", "projects"] | <h2 id="roget-thesaurus-parser">Roget Thesaurus parser</h2>
<p class="publish_date">
26 February 2015
</p>
<p>The Roget thesaurus does not seem to be widely used in natural language processing, for example the <a href="https://www.youtube.com/watch?v=nfoudtpBV68&list=PL6397E4B26D00A269">Stanford NLP course</a> does not even mention it at all; instead they all seem to focus on the <a href="http://wordnetweb.princeton.edu/perl/webwn">Wordnet</a> thesaurus.</p>
<p>I think that the Roget thesaurus has a lot to offer here:</p>
<ul>
<li>the thesaurus is in the public domain, so it can be used for free</li>
<li>The ontology/classification of Roget is more shallow than that of Wordnet, i think that would makes it easier to work with</li>
<li>In Roget thesaurus the word senses are grouped around Head words; each of these head words is also classified within the ontology</li>
</ul>
<p>The structure of the Roget thesaurus would be of great for such task as finding word similarities; two words or expressions that share the same head word would count as similar words. (Correction: it appears that I am not the first one to think of that; the following paper does that too - ‘Roget’s Thesaurus and Semantic Similarity’ by Mario JARMASZ and Stan SZPAKOWICZ <a href="http://arxiv.org/ftp/arxiv/papers/1204/1204.0245.pdf">here</a> )</p>
<p>In this project i have written a module in the python program language that parses the text of the Roget thesaurus, it also creates a data structure that represents the Roget thesaurus in memory. This data structure tries to preserve all information that is contained in the text of the Roget Thesaurus, it allows a program to look up the word senses and to navigate through the ontology.</p>
<p>The text that is parsed is based on the text of the Roget Thesaurus from archive.org ( <a href="https://archive.org/details/rogetsthesauruso10681gut">here</a> ) I am not quite sure as to which edition of the Roget thesaurus that is; the text does not mention it, however it mentions words like ‘Nuclear energy’ and ‘Huggies’, so it must be from the 20th century.</p>
<h3 id="obtaining-the-program">Obtaining the program</h3>
<p>you can obtain the module from pypi</p>
<p><code>
sudo pip install roget
</code></p>
<p>quite easy if it works.</p>
<p>An alternative is to get the sources from the github repository:</p>
<p><code>
git clone https://github.com/MoserMichael/cstuff.git
</code></p>
<p>One can then install it from the sources</p>
<p><code></code></p>
<p>cd cstuff/python-pypi/roget</p>
<p>on unix/linux:
sudo python setup.py install</p>
<p>on windows:
python setup.py install
</code></p>
<p>The html reference for the roget.py python module is <a href="http://mosermichael.github.io/cstuff/docs/html/python/roget.html">here</a></p>
<h3 id="the-roget-thesaurus">the Roget Thesaurus</h3>
<p>The Thesaurus classifies word senses into the following ontology</p>
<p>Each category is represented by a Node of type Category, this hierarchy is represented by a tree structure of nodes, each of them is of type Category.</p>
<blockquote>
<code>
<pre>
Category - root ()
Category - CLASS I (WORDS EXPRESSING ABSTRACT RELATIONS)
Category - SECTION I. (EXISTENCE)
Category (1. BEING, IN THE ABSTRACT)
Category (2. BEING, IN THE CONCRETE)
Category (3. FORMAL EXISTENCE)
Category (Internal conditions)
Category (External conditions)
Category (4. MODAL EXISTENCE)
Category (Absolute)
Category (Relative)
Category - SECTION II. (RELATION)
Category (1. ABSOLUTE RELATION)
Category (2. CONTINUOUS RELATION)
Category (3. PARTIAL RELATION)
Category (4. GENERAL RELATION)
Category - SECTION III. (QUANTITY)
Category (1. SIMPLE QUANTITY)
Category (2. COMPARATIVE QUANTITY)
Category (1 QUANTITY BY COMPARISON WITH A STANDARD)
Category (2 QUANTITY BY COMPARISON WITH A SIMILAR OBJECT)
Category (3 CHANGES IN QUANTITY)
Category (3. CONJUNCTIVE QUANTITY)
Category (4. CONCRETE QUANTITY)
Category - SECTION IV. (ORDER)
Category (1. ORDER IN GENERAL)
Category (2. CONSECUTIVE ORDER)
Category (3. COLLECTIVE ORDER)
Category (4. DISTRIBUTIVE ORDER)
Category (5. ORDER AS REGARDS CATEGORIES)
Category - SECTION V. (NUMBER)
Category (1. NUMBER, IN THE ABSTRACT)
Category (2. DETERMINATE NUMBER)
Category (3. INDETERMINATE NUMBER)
Category - SECTION VI. (TIME)
Category (1. ABSOLUTE TIME)
Category (2. RELATIVE TIME)
Category (1. Time with reference to Succession)
Category (3. RECURRENT TIME)
Category - SECTION VII. (CHANGE)
Category (1. SIMPLE CHANGE)
Category (2. COMPLEX CHANGE)
Category (Present Events)
Category (Future Events)
Category - SECTION VIII. (CAUSATION)
Category (1. CONSTANCY OF SEQUENCE IN EVENTS)
Category (2. CONNECTION BETWEEN CAUSE AND EFFECT)
Category (3. POWER IN OPERATION)
Category (4. INDIRECT POWER)
Category (5. COMBINATIONS OF CAUSES)
Category - CLASS II (WORDS RELATING TO SPACE)
Category - SECTION I. (SPACE IN GENERAL)
Category (1. ABSTRACT SPACE)
Category (2. RELATIVE SPACE)
Category (3. EXISTENCE IN SPACE)
Category - SECTION II. (DIMENSIONS)
Category (1. GENERAL DIMENSIONS)
Category (2. LINEAR DIMENSIONS)
Category (3. CENTRICAL DIMENSIONS)
Category (1. General.)
Category (2. Special)
Category - SECTION III. (FORM)
Category (1. GENERAL FORM)
Category (2. SPECIAL FORM)
Category (3. SUPERFICIAL FORM)
Category - SECTION IV. (MOTION)
Category (1. MOTION IN GENERAL)
Category (2. DEGREES OF MOTION)
Category (3. MOTION CONJOINED WITH FORCE)
Category (4. MOTION WITH REFERENCE TO DIRECTION)
Category - CLASS III (WORDS RELATING TO MATTER)
Category - SECTION I. (MATTER IN GENERAL)
Category - SECTION II. (INORGANIC MATTER)
Category (1. SOLID MATTER)
Category (2. FLUID MATTER)
Category (3. IMPERFECT FLUIDS)
Category - SECTION III. (ORGANIC MATTER)
Category (1. VITALITY)
Category (1. Vitality in general)
Category (2. Special Vitality)
Category (2. SENSATION)
Category (1. Sensation in general)
Category (2. Special Sensation)
Category ((1) Touch)
Category ((2) Heat)
Category ((4) Odor)
Category ((5) Sound)
Category ((i) SOUND IN GENERAL)
Category ((ii) SPECIFIC SOUNDS)
Category ((iii) MUSICAL SOUNDS)
Category ((iv) PERCEPTION OF SOUND)
Category ((6) Light)
Category ((i) LIGHT IN GENERAL)
Category ((ii) SPECIFIC LIGHT)
Category (Primitive Colors)
Category (Complementary Colors)
Category ((iii) PERCEPTIONS OF LIGHT)
Category - CLASS IV (WORDS RELATING TO THE INTELLECTUAL FACULTIES)
Category - DIVISION I (FORMATION OF IDEAS)
Category - SECTION I. (OPERATIONS OF INTELLECT IN GENERAL)
Category - SECTION II. (PRECURSORY CONDITIONS AND OPERATIONS)
Category - SECTION III. (MATERIALS FOR REASONING)
Category (Evidence for, against, and in mitigation)
Category - SECTION IV. (REASONING PROCESSES)
Category - SECTION V. (RESULTS OF REASONING)
Category - SECTION VI (EXTENSION OF THOUGHT)
Category (1. To the Past)
Category - SECTION VII. (CREATIVE THOUGHT)
Category - DIVISION II (COMMUNICATION OF IDEAS)
Category - SECTION I. (NATURE OF IDEAS COMMUNICATED)
Category - SECTION II. (MODES OF COMMUNICATION)
Category - SECTION III. (MEANS OF COMMUNICATING IDEAS)
Category (1. NATURAL MEANS)
Category (2. CONVENTIONAL MEANS)
Category (1. Language generally)
Category (2. Spoken Language)
Category (3. Written Language)
Category - CLASS V (WORDS RELATING TO THE VOLUNTARY POWERS)
Category - DIVISION I (INDIVIDUAL VOLITION)
Category - SECTION I. (VOLITION IN GENERAL)
Category (1. Acts of Volition)
Category (2. Causes of Volition)
Category (3. Objects of Volition)
Category - SECTION II. (PROSPECTIVE VOLITION)
Category (1. CONCEPTIONAL VOLITION)
Category (2. SUBSERVIENCE TO ENDS)
Category (1. Actual Subservience)
Category (3. PRECURSORY MEASURES)
Category - SECTION III. (VOLUNTARY ACTION)
Category (1. Simple voluntary Action)
Category (2. Complex Voluntary Action)
Category - SECTION IV. (ANTAGONISM)
Category (1. Conditional Antagonism)
Category (2. Active Antagonism)
Category - SECTION V. (RESULTS OF VOLUNTARY ACTION)
Category - DIVISION II (INTERSOCIAL VOLITION)
Category - SECTION I. (GENERAL INTERSOCIAL VOLITION)
Category - SECTION II. (SPECIAL INTERSOCIAL VOLITION)
Category - SECTION III. (CONDITIONAL INTERSOCIAL VOLITION)
Category - SECTION IV. (POSSESSIVE RELATIONS)
Category (1. Property in general)
Category (2. Transfer of Property)
Category (3. Interchange of Property)
Category (4. Monetary Relations)
Category - CLASS VI (WORDS RELATING TO THE SENTIENT AND MORAL POWERS)
Category - SECTION I. (AFFECTIONS IN GENERAL)
Category - SECTION II (PERSONAL AFFECTIONS)
Category (1. PASSIVE AFFECTIONS)
Category (2. DISCRIMINATIVE AFFECTIONS)
Category (3. PROSPECTIVE AFFECTIONS)
Category (4. CONTEMPLATIVE AFFECTIONS)
Category (5. EXTRINSIC AFFECTIONS)
Category - SECTION III. (SYMPATHETIC AFFECTIONS)
Category (1. SOCIAL AFFECTIONS)
Category (2. DIFFUSIVE SYMPATHETIC AFFECTIONS)
Category (3. SPECIAL SYMPATHETIC AFFECTIONS)
Category (4. RETROSPECTIVE SYMPATHETIC AFFECTIONS)
Category (1. MORAL OBLIGATIONS)
Category (2. MORAL SENTIMENTS)
Category (3. MORAL CONDITIONS)
Category (4. MORAL PRACTICE)
Category (5. MORAL INSTITUTIONS)
Category (1. SUPERHUMAN BEINGS AND REGIONS)
Category (2. RELIGIOUS DOCTRINES)
Category (3. RELIGIOUS SENTIMENTS)
Category (4. ACTS OF RELIGION)
Category (5. RELIGIOUS INSTITUTIONS)
</pre>
</code>
</blockquote>
<p>In this tree there or leaf categories (an example of a leaf category is ‘1. BEING, IN THE ABSTRACT’) the next level are head words;
Word senses cluster around this head word.</p>
<blockquote>
<code>
<pre>
Category - root ()
Category - CLASS I (WORDS EXPRESSING ABSTRACT RELATIONS)
Category - SECTION I. (EXISTENCE)
Category (1. BEING, IN THE ABSTRACT)
#1 Headword (Existence)
#2 Headword (Inexistence)
Category (2. BEING, IN THE CONCRETE)
#3 Headword (Substantiality)
#4 Headword (Unsubstantiality)
Category (3. FORMAL EXISTENCE)
Category (Internal conditions)
#5 Headword (Intrinsicality)
Category (External conditions)
#6 Headword (Extrinsicality)
Category (4. MODAL EXISTENCE)
Category (Absolute)
#7 Headword (State)
Category (Relative)
#8 Headword (Circumstance)
Category - SECTION II. (RELATION)
Category (1. ABSOLUTE RELATION)
#9 Headword (Relation)
#10 Headword (Irrelation)
#11 Headword (Consanguinity)
#12 Headword (Correlation)
#13 Headword (Identity)
#14 Headword (Contrariety)
#15 Headword (Difference)
. . . (lots more )
</pre>
</code>
</blockquote>
<p>Here is one example head word from the text of the Thesaurus.</p>
<blockquote>
<code>
<pre>
CLASS I
WORDS EXPRESSING ABSTRACT RELATIONS
SECTION I.
EXISTENCE
1. BEING, IN THE ABSTRACT
1. Existence -- N. existence, being, entity, ens [Lat.], esse [Lat.],
subsistence.
reality, actuality; positiveness &c adj.; fact, matter of fact,
sober reality; truth &c 494; actual existence.
presence &c (existence in space) 186; coexistence &c 120.
stubborn fact, hard fact; not a dream &c 515; no joke.
center of life, essence, inmost nature, inner reality, vital
principle [Science of existence], ontology.
V. exist, be; have being &c n.; subsist, live, breathe, stand, obtain,
be the case; occur &c (event) 151; have place, prevail; find oneself,
pass the time, vegetate.
consist in, lie in; be comprised in, be contained in, be
constituted by.
come into existence &c n.; arise &c (begin) 66; come forth &c
(appear) 446.
become &c (be converted) 144; bring into existence &c 161.
abide, continue, endure, last, remain, stay.
Adj. existing &c v.; existent, under the sun; in existence &c n.;
extant; afloat, afoot, on foot, current, prevalent; undestroyed.
real, actual, positive, absolute; true &c 494; substantial,
substantive; self-existing, self-existent; essential.
well-founded, well-grounded; unideal, unimagined; not potential &c
2; authentic.
Adv. actually &c adj.; in fact, in point of fact, in reality; indeed;
de facto, ipso facto.
Phr. ens rationis [Lat.]; ergo sum cogito [Lat.], thinkest thou
existence doth depend on time? [Byron].
</pre>
</code>
</blockquote>
<p>let us look at the head word Existence ; after the head words there is a set of word senses, where each word sense is separated by a comma or semicolon.
it seems that more closely related word senses are separated by comma, so the data structure will group these closely related senses under a Sense node
More loosely related word senses are separated by semicolon.</p>
<p>In the data structure each set of comma separated word senses has its own parent node of type SenseGroup;
if a single word sense is followed by a semicolon then this is represented by a node that is placed directly under the head word node.</p>
<p>Note that some word senses have optional links to other head words.</p>
<p>Now here is a text report that describes this structure:</p>
<blockquote>
<code>
<pre>
Category - root ()
Category - CLASS I (WORDS EXPRESSING ABSTRACT RELATIONS)
Category - SECTION I. (EXISTENCE)
Category (1. BEING, IN THE ABSTRACT)
#1 Headword (Existence)
SenseGroup ()
Sense (existence) /N/
Sense (being)
Sense (entity)
Sense (ens) comment: Lat.
Sense (esse) comment: Lat.
Sense (subsistence)
SenseGroup ()
Sense (reality)
Sense (actuality)
Sense (positiveness) /Adj/
SenseGroup ()
Sense (fact)
Sense (matter of fact)
Sense (sober reality)
Sense (truth) [link: #494 (Truth) ]
Sense (actual existence)
Sense (presence) [link: #186 (Presence) ]
Sense (coexistence) [link: #120 (Synchronism) ]
SenseGroup ()
Sense (stubborn fact)
Sense (hard fact)
Sense (not a dream) [link: #515 (Imagination) ]
Sense (no joke)
SenseGroup ()
Sense (center of life)
Sense (essence)
Sense (inmost nature)
Sense (inner reality)
Sense (vital principle) comment: Science of existence
Sense (ontology)
SenseGroup ()
Sense (exist) /V/
Sense (be)
Sense (have being) /N/
SenseGroup ()
Sense (subsist)
Sense (live)
Sense (breathe)
Sense (stand)
Sense (obtain)
Sense (be the case)
Sense (occur) [link: #151 (Eventuality) ]
SenseGroup ()
Sense (have place)
Sense (prevail)
SenseGroup ()
Sense (find oneself)
Sense (pass the time)
Sense (vegetate)
SenseGroup ()
Sense (consist in)
Sense (lie in)
SenseGroup ()
Sense (be comprised in)
Sense (be contained in)
Sense (be constituted by)
Sense (come into existence) /N/
Sense (arise) [link: #66 (Beginning) ]
Sense (come forth) [link: #446 (Visibility) ]
Sense (become) [link: #144 (Conversion) ]
Sense (bring into existence) [link: #161 (Production) ]
SenseGroup ()
Sense (abide)
Sense (continue)
Sense (endure)
Sense (last)
Sense (remain)
Sense (stay)
Sense (existing) /Adj/
SenseGroup ()
Sense (existent)
Sense (under the sun)
Sense (in existence) /N/
Sense (extant)
SenseGroup ()
Sense (afloat)
Sense (afoot)
Sense (on foot)
Sense (current)
Sense (prevalent)
Sense (undestroyed)
SenseGroup ()
Sense (real)
Sense (actual)
Sense (positive)
Sense (absolute)
Sense (true) [link: #494 (Truth) ]
SenseGroup ()
Sense (substantial)
Sense (substantive)
SenseGroup ()
Sense (self-existing)
Sense (self-existent)
Sense (essential)
SenseGroup ()
Sense (well-founded)
Sense (well-grounded)
SenseGroup ()
Sense (unideal)
Sense (unimagined)
Sense (not potential) [link: #2 (Inexistence) ]
Sense (authentic)
Sense (actually) /Adv/
SenseGroup ()
Sense (in fact)
Sense (in point of fact)
Sense (in reality)
Sense (indeed)
SenseGroup ()
Sense (de facto)
Sense (ipso facto)
Sense (ens rationis) /Phr/ comment: Lat.
SenseGroup ()
Sense (ergo sum cogito) comment: Lat.
Sense (thinkest thou existence doth depend on time?) comment: Byron
</pre>
</code>
</blockquote>
<h3 id="the-programming-interface">The programming interface</h3>
<p>The html reference for the roget.py python module is <a href="http://mosermichael.github.io/cstuff/docs/html/python/roget.html">here</a></p>
<p>In order to create an instance of the Roget Thesaurus class first construct an instance of RogetBuilder and then call the method parse - please note that the file 10681-body.txt must be in the same directory as the script roget.py .</p>
<blockquote>
<code>
<pre>
import roget
builder = roget.RogetBuilder( 1 )
rogetThesaurus = builder.parse()
</pre>
</code>
</blockquote>
<p>The following function will look up a word in the index of this class; the word sense index is mapping the text of the word sense to a list of reference - each reference refers to the word sense as it appears in the Roget thesaurus. Given the node we can examine the node properties and navigate to the parent node, etc.</p>
<blockquote>
<code>
<pre>
def lookupWord( rogetThesaurus, word ):
idx = rogetThesaurus.senseIndex
wordSenses = idx[ word ]
if wordSenses != None:
for s in wordSenses:
while s != None:
print s.toString(), " / ",
s = s.parent
print "\n"
else:
print "no word senses found\n"
lookupWord(rogetThesaurus, "fact")
lookupWord(rogetThesaurus, "fiction")
lookupWord(rogetThesaurus, "love")
lookupWord(rogetThesaurus, "hate")
</pre>
</code>
</blockquote>
<h3 id="computing-the-semantic-similarities-between-two-terms">Computing the semantic similarities between two terms;</h3>
<p>class RogetThesaurus has a method for computing a score for the semantic similarity of two terms;</p>
<p>The algorithm used by this function is as follows:
produce an array of nodes for each search term
for each node in the sense index
add the node and its parent nodes (until reaching the first category node)
if the node has a link to another node, then also add the link target and its parent nodes
sort the array by internal node id (each node in the ontology has its unique node id)</p>
<p>merge the two arrays: if a node with identical unique node id is found then
if the node is a sense group then report a score of 100
if the node is a head word then set the best score to 90
if the node is a category then set the best score to 80</p>
<p>return the best score</p>
<p>This algorithm is implemented by the following function:</p>
<blockquote>
<code>
<pre>
# add all parent nodes to array (up to first category); adds the
def _semHelpAddParents( self, s, ret):
while s != None:
ret.append( s )
if s.type == ROGET_NODE_CATEGORY:
break
s = s.parent
def _semHelpSortedSet( self, word ):
senses = self.senseIndex
wordSenses = senses[ word ]
ret = []
if wordSenses != None:
for s in wordSenses:
self._semHelpAddParents( s, ret )
if s.link != None:
self._semHelpAddParents( s.link, ret )
ret.sort( key=lambda elm: elm.internalId ) # sort by internal id
#print "[",
#for s in ret:
# print s.internalId, "," ,
#print "]"
return ret
def semanticSimilarity( self, seq1, seq2 ):
''' computes the semantic similarity between two terms,
returns the following tuple (similarity-score, common-node-in-roget-thesaurus)
the similarity score:
100 - both terms appear in the same SenseGroup node
90 - both terms he the same head word
80 - both terms appear in the same leaf category
0 - everything else
common-node-in-roget-thesaurus: is None if the score is 0;
otherwise it is the common node that the score is based on
'''
arr1 = self._semHelpSortedSet( seq1 )
arr2 = self._semHelpSortedSet( seq2 )
# merging two sorted arrays
score = 0
rnode = None
pos1 = 0
pos2 = 0
if arr1 == None or arr2 == None:
return (0, None)
while pos1 < len( arr1 ) and pos2 < len( arr2 ):
if arr1[ pos1 ].internalId == arr2[ pos2 ].internalId:
node = arr1[ pos1 ]
if node.type == ROGET_NODE_CATEGORY:
nscore = 80
elif node.type == ROGET_NODE_HEADWORD:
nscore = 90
elif node.type == ROGET_NODE_SENSE_GROUP:
nscore = 100
if score < nscore:
score = nscore
rnode = node
if score == 100:
break
pos1 += 1
pos2 += 1
elif arr1[ pos1 ].internalId < arr2[ pos2 ].internalId:
pos1 += 1
else:
pos2 += 1
return (score, rnode)
</pre>
</code>
</blockquote>
<p>I could not evaluate accuracy of the results, as I don’t have the test data used by other studies/.</p>
<h3 id="acknowledgment">Acknowledgment</h3>
<p>Thanks to my father Ilja Moser for teaching me about the properties of the Roget Thesaurus.</p>
Python debugging tools
2015-02-24T00:00:00+00:00
http://mosermichael.github.com//all/projects/2015/02/24/python-backtrace
["", "projects"] | <h2 id="python-debugging-tools">Python debugging tools</h2>
<p class="publish_date">
24 February 2015
</p>
<p>I have been learning python lately; So far i would use Perl 5 as my scripting language of choice, but at work they start to phase out usage of Perl. This article presents some debugging tools that I wrote in the process</p>
<hr />
<h3 id="better-python-stack-trace">better python stack trace</h3>
<p>Learning a new programming language is always an aggravating business; I did not quite like is the format of the stack backtrack that python 2.7 produces when it encounters a run-time error.</p>
<p>So I wrote a module that produces a better backtrace that looks like the one that gdb produces with gdb commands ‘bt’ and ‘bt all’,</p>
<p>Here is an example stack trace:</p>
<blockquote>
<code>
<pre>
#1 def fact(n = 1) at test_pd.py:10
Local variables:
loc = 2
loc2 = [0]
Calls next frame at:
pd.print_stack_ex() at: test_pd.py:10
#2 def fact(n = 2) at test_pd.py:8
Local variables:
loc = 4
loc2 = [0, 1]
Calls next frame at:
return n * fact( n - 1 ) at: test_pd.py:8
#3 def fact(n = 3) at test_pd.py:8
Local variables:
loc = 6
loc2 = [ 0, 1, 2 ]
Calls next frame at:
return n * fact( n - 1 ) at: test_pd.py:8
#4 def fact(n = 4) at test_pd.py:8
Local variables:
loc = 8
loc2 = [ 0, 1, 2, 3 ]
Calls next frame at:
return n * fact( n - 1 ) at: test_pd.py:8
#5 def main() at test_pd.py:36
Local variables:
Calls next frame at:
print fact(4) at: test_pd.py:36
#6 def <module>() at test_pd.py:53
Calls next frame at:
main() at: test_pd.py:53
</pre>
</code>
</blockquote>
<p>This module is called pd and that stands for python debugging module. It is similar to the standard traceback module that comes with the python libraries.</p>
<p>The pd module also exports the die and die2 functions, these print an error message and exit the program, they also print out a stack trace (die2 with a more detailed stack that prints out object references)</p>
<p>So these are similar to the built in die function in Perl;</p>
<h3 id="how-to-install-the-pd-module">how to install the pd module</h3>
<p>you can obtain the module from pypi</p>
<p><code>
sudo pip install pd
</code></p>
<p>quite easy if it works.</p>
<p>An alternative is to get the sources from the github repository:</p>
<p><code>
git clone https://github.com/MoserMichael/cstuff.git
</code></p>
<p>One can then install it from the sources</p>
<p><code></code></p>
<p>cd cstuff/python-pypi/pd</p>
<p>on unix/linux:
sudo python setup.py install</p>
<p>on windows:
python setup.py install
</code></p>
<h3 id="python-tracing-with-ntrace">python tracing with ntrace</h3>
<p>The python program trace.py runs a program and displays source lines before they are executed (it can also do basic coverage report, but my ntrace.py module does only tracing)</p>
<p>The ntrace.py program presented here can also print out additional information:</p>
<ul>
<li>it can show function arguments and return values</li>
<li>it can display variables when they are modified</li>
<li>it can display the value of accessed variables</li>
</ul>
<p>(Note: the ntrace program needs an extension module for part of its functionality; i have tested that on python2.7 only)</p>
<h3 id="how-to-install-the-pd-module-1">how to install the pd module</h3>
<p>First check that the python headers are installed on the system; for debian based systems that is (this program has its own extension module, so installation wiill compile the module)</p>
<p><code>
apt-get install libpython2.7-dev
</code></p>
<p>also you would need a c compiler installed on the system</p>
<p><code>
apt-get install gcc
</code></p>
<p>now you can obtain the module from pypi</p>
<p><code>
sudo pip install ntrace
</code></p>
<p>quite easy if it works.</p>
<p>An alternative is to get the sources from the github repository:</p>
<p><code>
git clone https://github.com/MoserMichael/cstuff.git
</code></p>
<p>One can then install it from the sources</p>
<p><code></code></p>
<p>cd cstuff/python-pypi/ntrace</p>
<p>on unix/linux:
sudo python setup.py install</p>
<p>on windows:
python setup.py install
</code></p>
<h3 id="how-to-use-the-tracer">how to use the tracer</h3>
<p>after installing the module you can invoke the command line program as a module</p>
<p><code></code></p>
<p>python -m ntrace/trace</p>
<p></code></p>
<p>this will give you the following help message:</p>
<blockquote>
<code>
<pre>
Usage: /usr/local/lib/python2.7/dist-packages/ntrace-0.0.1-py2.7-linux-i686.egg/ntrace/trace.py [OPTIONS] <file> [ARGS}
Traces execution of python program;
By default it shows function parameters and function return values
Options:
-f trace function parameter calls (default: on)
-F do not trace function parameters
-r trace function return values (default: on)
-R do not trace function return values
-l trace source lines before they are executed (default: on)
-L do not trace source lines before they are executed
-s trace setting of variablesa (default: off)
-S do not trace setting of variables
-g trace getting of variables (default: off)
-G do not trace getting of variables
-o print out object content (default: off)
-O do not print out object content
-t show the current thread id in prefix for each source line (default on)
-T hide the current thread id
-n show the nesting level, how many levels down the call stack (default on)
-N hide the nesting level
By default the following options are on: -l -f -r
</pre>
</code>
</blockquote>
<p>The functionality can also be accessed from an class: here the Ntracer class is used to run the program test_ntrace2.py with tracing.</p>
<blockquote>
<code>
<pre>
import ntrace
t = ntrace.NTracer(ntrace.TO_FUNCTION_ARGUMENTS | ntrace.TO_FUNCTION_RVALUES | ntrace.TO_TRACE_SRC_LINES )
t.trace( 'test_ntrace2.py' , False)
</pre>
</code>
</blockquote>
<h3 id="how-it-works">how it works</h3>
<p>One can start another python program; first the build in python function <a href="https://docs.python.org/2.7/library/functions.html#compile">compile</a> is used to compile source code has to be compiled into an abstract syntax tree (AST); next the <a href="https://docs.python.org/2/reference/simple_stmts.html#grammar-token-exec_stmt">exec statement</a> takes this internal representation and runs it.</p>
<p>Python debuggers (and tracers) add the installation of a callback function between these two step by means of <a href="https://docs.python.org/2/library/sys.html#sys.settrace">sys.settrace</a> ( it is installed prior to invoking the exec statement); the supplied callback is called upon a python function call and prior to returning from a function.
leaving</p>
<p>How then can we trace the assignments of variables in python?
For access to global variables this is relatively easy: the exec statement returns a dictionary that represents the global namespace - setting a global variable with the name aaaa will result into insertion of the key ‘aaaa’ into this dictionary. Luckily we can subclass a custom dictionary class, this subclass that will override insertion and lookup of entries. The overriden insertion/lookup function are now able to trace global variable modification/access.</p>
<p>Local variables are more tricky: prior to calling python method the python function, the python interpreter must create a stack frame object . This stack frame object has a dictionary for the local namespace; again, setting a local variable with the name bbbb will result into insertion of the key ‘bbbb’ into this dictionary that is owned by the frame object. The solution for local variables will again substitute the python dictionary with an object that is derived from class Dict.</p>
<p>Unfortunately we can’t modify or substitute this dictionary from python code, but of course we can do so from a native library. That’s exactly what is being done by the ntrace package.</p>
<p>(PS: Personally I think that it is a bit wasteful that Python is handling all variable accesses as hash lookup by variable name; for example the <a href="http://mosermichael.github.io/cstuff/all/pooh-lan/2012/12/11/pooh-intro.html">Pooh programming language</a> translates variables into indexes on the stack/heap objects. this use of hash table for variable lookup is really inefficent, i find it quite amazing. Just prooves again that speed is not one of the priorities of scripting languages; people are ready to trade that for an improvement of preceived expressiveness)</p>
<hr />
<h3 id="comparing-perl-and-python">comparing Perl and Python</h3>
<p>Now here comes some subjective ranting on the subject of Perl vs. python.</p>
<p>I did like the following in python:</p>
<ul>
<li>it is not allowed to access an undefined variable (unlike ruby where this is allowed);</li>
<li>there is lots of support for python on sites like <a href="http://stackoverflow.com">stack overflow</a> ; whenever i had a question there was an answer ready.</li>
<li>also lots of libraries are available in python, and using the pypi installer is quite easy.</li>
<li>doing classes in python is mostly intuitive</li>
<li>in python every object is a reference, (in Perl you have values and references); i guess that makes it easier to use.</li>
</ul>
<p>here are some bad parts of python and things i miss from Perl; in any event it took me some time to get used to these features.</p>
<ul>
<li>the syntax of python is a bit aggravating - writing a colon at the end of some statements (like if, for, def, try, etc); the problem is that this is not a universal feature - assignments do not need a colon at the end, so i often omit the colon by error when it is needed.</li>
<li>when an undefined (not assigned) variable is used this is a run time error, this means that the program has to run until it encounters this error; in Perl Strict mode this is a check that is done while parsing the script, so that these kinds of errors are reported much earlier.</li>
<li>when appending an integer value to a string we do have to convert it to string format with the str built in.</li>
<li>in PERL i miss the way that online documentation is structured; especially perlfunc <a href="http://perldoc.perl.org/perlfunc.html#Perl-Functions-by-Category">here</a> where links to specific functions are clustered around the way (similar to headwords in Roget thesaurus)</li>
<li>i am sceptic about exceptions in scripting languages - i still think that doing exceptions for scripting is an overkill.</li>
<li>the way they do getter and setter functions in classes by means of property declarations is a bit non intuitive</li>
<li>Perl has a better integration of regular expressions into the languages; in python we have to get the match object, and then extract the match groupings; in Perl you just write $1 and there you have the sub group.</li>
<li><a href="http://www.cpan.org/">cpan</a> is better organized than the <a href="https://pip.pypa.io/">pip</a> archive.</li>
<li>in Perl they have references, i guess that makes it less easy to use, still this feature is powerful.</li>
</ul>
<p>it seems to be a longer term trend: each generation of programmers has a favorite scripting language, and so I guess that in another twenty years we will see a different prevalent scripting language.</p>
<p>Lets hope i will still be around to witness the next change. I guess that this future scripting language will also have its good points and also have some aggravating details.</p>
Natural language processing revisited
2015-02-05T00:00:00+00:00
http://mosermichael.github.com//all/blog/2015/02/05/nlp-revisited
["", "blog"] | <h2 id="natural-language-processing-revisited">Natural language processing revisited</h2>
<p class="publish_date">
5 February 2015
</p>
<p>There is a very interesting course on Natural language processing on youtube, it is by Prof. Dan Jurafsky & Chris Manning.</p>
<iframe width="560" height="315" src="//www.youtube-nocookie.com/embed/nfoudtpBV68?list=PL6397E4B26D00A269" frameborder="0" allowfullscreen=""></iframe>
<p>i took some notes while watching these lectures (they are here at the end of this article);</p>
<p>Now some twenty years ago i attended a course on natural language processing and so to me it was very interesting to see how the field has progressed.</p>
<p>Things that changed are</p>
<ul>
<li>renewed interest in dependency grammars</li>
<li>strong focus on commercial applications, previously the focus was on demonstration systems; demonstration systems are build with the limited purpose of demonstrating the feasibility of a concept, so they are much less robust than commercial applications ;</li>
<li>nowadays there is a strong interest in models that are based on probability</li>
<li>corpus based approach to evaluation of result</li>
</ul>
<p>Natural language processing is a very interesting field of study as it combines some linguistics, statistics/information theory/machine learning, programming and AI tricks.</p>
<h4 id="interest-in-dependency-grammars">interest in dependency grammars</h4>
<p>For a very long time, up until the late nineties, there was this strong focus on Context free constituency grammars - these are similar to BNF rules that define the syntax of a programming
languages: in both cases the inner node of a parse tree is just the name of a abstract category - for a given sentence, the type of inner node is determined by the production rule that can be
applied to parse the sentence; the inner nodes of the parse three knows nothing about the leafs of the tree.</p>
<p>The problem with this approach is that it produces very deep tree structures and it is very hard for a program to assign a meaning to such a complex structure; Another point is that the grammar
grows quite complex as it has to be able to describe all possible cases of correct utterances, a complex grammar makes an even more complex parse tree; also this complexity of grammar turns parsing
into a challenging and time consuming task; Still another problem: it is very hard to formulate a grammar for languages with relatively free word order.</p>
<p>Nowadays there is a renewed focus on dependency grammars; (see lectures <a href="https://www.youtube.com/watch?v=UTnHwzVAIOo&index=72&list=PL6397E4B26D00A269">here</a> and <a href="https://www.youtube.com/watch?v=ZnW3yet8ngo&list=PL6397E4B26D00A269&index=73">here</a> and <a href="https://www.youtube.com/watch?v=Lm00hWPmdTQ&index=74&list=PL6397E4B26D00A269">here</a> ) Here instead of abstract categories without context there are head words (with the main verb ending up as the highest level node)</p>
<p>The result is a more shallow parse tree; for a program this makes it much is easier to interpret such a structure: a head verb can be mapped to a function that does an action, all subordinate
clauses are then just argument to the procedure that stands for this action.</p>
<p>(incidentally context free grammars evolved into this direction: in modern context free grammars the categories are annotated with the head word: (see x-bar theory and lexicalized context free
grammars), however the parse tree here remains more complex, even in its modern form</p>
<p>Now this is not quite new; the <a href="http://hci.stanford.edu/~winograd/shrdlu/">SHRDLU</a> program written in 1972 by Terry Winograd did have a similar structure: (<a href="http://dspace.mit.edu/handle/1721.1/7095">here</a> is Winograd’s dissertation that describes this program)</p>
<p>The user of the program enters phrases in English that talk about geometric objects on a table: like here</p>
<p>Like this one</p>
<p><img src="/cstuff/images/shrdlu-1.png" alt="" /></p>
<p>The program creates a database entry for each geometric object that is mentioned, this database bentry lists properties of each object, like color, shape, position of the object relative to other objects
on the table. It can change the properties of its database entries according to the requests that are typed in in English; it also answers question in English .
It is also drawing a picture that contains all of the objects that it knows about. In the picture thereis also an arrow that points towards the block that the program is ‘holding’)</p>
<p>The input sentence is parsed according to a systemic grammar; this grammar also produces a relatively shallow parse tree where the verbs are directly mapped to procedures that handle the input text;</p>
<p>However this program was ignored by the linguists of MIT; practical considerations that were required to build computer systems, like depth of the parse tree, were of not interests to them.</p>
<h4 id="strong-focus-on-commercial-applications-less-focus-on-preset-notions">strong focus on commercial applications less focus on preset notions</h4>
<p>Once upon a time artificial intelligence research was funded exclusively by the US military, by means of programs like ARPA/DARPA. The ‘golden days’ of artificial intelligence were in the sixties to mid-seventies, when US research spending was were very high in general because of the space race.</p>
<p>Research was centered around a few university departments at top universities like MIT, Stanford etc; Each research department was led by a strong personality who would organize the funding and
who could formulate a general approach to his area of studies (like the “Syntactic Structures” of Noam Chomsky)</p>
<p>Terry Winograd in this interview <a href="http://conservancy.umn.edu/bitstream/handle/11299/107717/oh237taw.pdf?sequence=1&isAllowed=y">here</a></p>
<blockquote>
TW: And of course at MIT, there was a huge amount of syntax, but that was another one of these sort of feuding worlds; you couldn't talk to Chomsky and Minsky both, it was impossible. The one contact I had is I went and took a course from Chomsky, wrote a term paper saying why I thought this AI-ish approach would work, and he gave me a 'C' on the term paper. That was the end of my contact with the Linguistics Department.
</blockquote>
<p>Also <a href="http://openarchive.cbs.dk/bitstream/handle/10398/6460/2002-7.pdf?sequence=1">here</a></p>
<blockquote>
TW: Minsky and Chomsky were both very strong personalities. You could go to one or the other but you couldn't be in the middle. That is how MIT is, it has strong boundaries between its departments.
MV: But you were allowed to do computers and linguistics?
TW: Minsky wanted me to do linguistics and prove that his students could do better linguistics than Chomsky's. I was not the first. Dave Waltz, who eventually went on to do work in vision, was
doing a research project on linguistics when I got there, as was Gene Charniak. And in the prior generation Daniel Bobrow and Bertram Raphael had both done language projects. So it was a major
stream of work in the AI Lab."
</blockquote>
<p>One would have expected that this setting would lead to personal fiefdoms, one of the features of such an organization is that everyone is protecting his own turf and not eager to cooperate.
Now what they had at MIT’s AI lab was just the opposite; Researchers had a lot of autonomy with regards to their projects; the people who were in esteem where the programmers/Lisp hackers; also they developed a culture of information sharing and cross boundary cooperation.</p>
<blockquote>
"WINOGRAD: It was people who were basically working on their own and then knowing enough about what each other was doing to join up. So, for example, this stuff on Micro-Planner was useful for all three of the people who did it, so we got together and did that. And to comment, and so on. I probably saw Marvin - well, he wasn't actually my official advisor, Seymour was, but I don't think I saw either of them to talk about my dissertation more than once every few months. It wasn't like they were directing the project."
</blockquote>
<p>another point is that the research departments were producing demonstration systems of simplified domains (like the block world in SHRDLU), not actual products:</p>
<blockquote>
Implementation meant something you could show off. It didn't mean something that somebody
else could use. So - I mean, my thesis was certainly that way. I mean, you know, the famous dialogue with SHRDLU where you could pick up a block, and so on, I very carefully worked through, line by line. If you sat down in front of it, and asked it a question that wasn't in the dialogue, there was some probability it would answer it. I mean, if it was reasonably close to one of the questions that was there in form and in content, it would probably get it. But there was no attempt to get it to the point where you could actually hand it to somebody and they could use it to move blocks around. And there was no pressure for that whatsoever. Pressure was for something you could demo. Take a recent example, Negroponte's Media Lab, where instead of "perish or publish" it's "demo or die." I think that's a problem. I think AI suffered from that a lot, because it led to "Potemkin villages", things which - for the things they actually did in the demo looked good, but when you looked behind that there wasn't enough structure to make it really work more generally.
</blockquote>
<p>This changed: the personal computer and wide spread adoption of the Internet created many commercial applications of natural language processing; the result was that a lot of focus shifted
towards the research divisions of Microsoft, IBM and Bells Labs.</p>
<p>These applications were:</p>
<ul>
<li>
<p>spelling checkers for word processors</p>
</li>
<li>
<p>spam detection for email systems</p>
</li>
<li>
<p>information retrieval systems for WEB search.</p>
</li>
<li>speech recognition</li>
<li>question answering / text classification /sentiment analysis / relationship extraction / etc. etc.</li>
</ul>
<p>I guess that these new requirements generated a shift in priorities: now the dominant explanation would be the one that could be used to build a working system, not the ones who would correspond to preset notions of how things were to be explained.</p>
<p>Also the natural language processing course that i did some twenty years ago was not very well attended, however i guess that this would have been different if students could have been able to predict that this field would create the biggest software company in the word.</p>
<h3 id="new-focus-on-probabilistic-models">New focus on probabilistic models</h3>
<p>Since the fifties probabilistic models were used in information retrieval, however in the study of grammar they were not very much used.</p>
<p>I guess this was due to opposition by Noam Chomsky;</p>
<p>Peter Norvig and Prof. Noam Chomsky are debating this issue <a href="http://norvig.com/chomsky.html">here</a></p>
<p>Chomsky’s argument is quoted as follows: “It’s true there’s been a lot of work on trying to apply statistical models to various linguistic problems. I think there have been some successes, but a lot of failures. There is a notion of success … which I think is novel in the history of science. It interprets success as approximating unanalyzed data.”</p>
<p>Chomsky argues that statistical models do not add insight;
Peter Norvig cites many examples where statistical meaning did add insight in physics and of course sites many successful systems that utilize this technique.</p>
<p>The debate does not mentioned the following: Chomsky established his approach as an opposition to previous behaviorist explanations of language acquisition; Skinner argues that children learn language by associating words with meaning; here good performance is encouraged by correct behavior (reinforcement principle); Chomsky argues that this stimulus cannot explain how a child manages to acquire language in such a short time; he argues that humans come with a built in facility of understanding language called universal grammar, so language acquisition is just a process of applying these general principles;</p>
<p>Now Chomsky is trying to identify and understand these Universal principles by means of introspective reasoning; Interesting, another possibility would have have been a more empirical approach: use statistics in order to study language acquisition patterns and use that in order to discern universal principles; Now the problem is that this is that statics are the favorite method of behaviorist psychologies, and Chomsky established his paradigm as opposition to this approach.</p>
<p>However nowadays it is more important to produce working systems, and theoretical considerations of this kind seem to be less important than that.</p>
<h3 id="corpus-based-approach-to-evaluation-of-result">corpus based approach to evaluation of result</h3>
<p>Now every algorithm is tuned over a hand classified text set (corpus). For parsers there is always a tree bank - a set of sentences that has been parsed into either a constituency tree or dependency graph. Part of this data is used as the training set - the set of data to tune the performance of an algorithm; the other part of this data is used for evaluation of performance.</p>
<p>One practical problem is that this data costs a lot of money - so they do maintain some artificial barriers of entry; however some of the data for some languages is available with an open source license (<a href="http://en.wikipedia.org/wiki/Treebank">see here</a> )</p>
<p>Another problem is that the selection of the test data may also constitute some form of bias on its own; for example parser treebanks are not that large because the task of hand parsing of the text is a bit complex; POS pre-tagged corpus is larger; as the task of hand tagging is easier.</p>
<hr />
<p>Lecture notes on Natural language processing course by Prof. Dan Jurafsky & Chris Manning.</p>
<p>(2 - 1 - Regular Expressions)</p>
<ul>
<li>uses regular expressions to parse tokens; uses http://regexpal.com to show regex syntax (Yawn).</li>
<li>
<p>edits regular expression to fix problems:</p>
</li>
<li>Increase accuracy/precision to exclude strings should that not have matched (False positive Type I)
<ul>
<li>Increase precision to include string we should have matched (False negatives - Type II)</li>
</ul>
</li>
<li>regular expression matching is often the first model of any text processing task; or as first stages for more complex processing.</li>
</ul>
<p>(2 - 2 - Regular Expressions in Practical NLP)</p>
<ul>
<li>English tokenizer in Stanford NLP library; written in jflex lexer. (java version of unix tool flex)</li>
</ul>
<p>covers lots of conventions: like:</p>
<p>(acronyms as regexes)
ABMONTH=Jan|Feb|March …</p>
<p>show regex for phone numbers (quite complicated - parse country extension, etc. lots of special cases, like regex for parsing email addresses)</p>
<p>(2 - 3 - Word Tokenization)</p>
<ul>
<li>
<p>Lemma: share same stem + part of speech (cat , cats)</p>
</li>
<li>
<p>word form: inflected ‘surface form’ - cat,cats are different word forms.</p>
</li>
<li>
<p>counting words: are fillers/fragments like Ah,Uh counted as words?
counting tokens? or number of dictionary entries? or number of lemma’s</p>
</li>
<li>
<table>
<tbody>
<tr>
<td>convention</td>
<td>N</td>
<td>- number of tokens</td>
<td>V</td>
<td>number of dictionary entries (vocabulary size)</td>
</tr>
</tbody>
</table>
</li>
<li>
<table>
<tbody>
<tr>
<td>how big is vocabulary of a corpus? Church&Gale estimate</td>
<td>V</td>
<td>> O (SQRT(</td>
<td>N</td>
<td>)</td>
</tr>
</tbody>
</table>
</li>
</ul>
<p>examples of fiddling with unix command pipelines:</p>
<table>
<tbody>
<tr>
<td>tr ~~sc ‘a-zA-Z’’\n’</td>
<td>sort</td>
<td>uniq # turn non letter characters to newlines (what about i’m~~> i + m) then sort then remove repetitions.</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>tr -sc ‘a-zA-Z’’\n’</td>
<td>sort</td>
<td>uniq</td>
<td>sort -n -r # now sorts by frequency</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>tr ‘A-Z’ ‘a-z’</td>
<td>tr -sc ‘a-zA-Z’’\n’</td>
<td>sort</td>
<td>uniq</td>
<td>sort -n -r # now does it case insensitive (turn to lower caps)</td>
</tr>
</tbody>
</table>
<p>other problems:</p>
<ul>
<li>
<p>now Finland’s -> Findland + s ; state-of-the-art -> state + of + the + art; m.p. -> m + p</p>
</li>
<li>
<p>compound word segmentation (German); French: L’ensemble</p>
</li>
<li>
<p>segmentation of tokens in Chinese+Japanese (no spaces)</p>
</li>
<li>
<p>(use matching of longest dictionary entry at cursor</p>
</li>
<li>this wouldn’t work for English without spaces - more ambiguous choices)).</li>
<li>Japanese + mix of different alphabets.</li>
</ul>
<p>(2 - 4 - Word Normalization and Stemming)</p>
<ul>
<li>
<p>‘normalize’ after tokenization - this means different things</p>
</li>
<li>for information retrieval: input query token must match the internalized representation of data in a program (example: delete punctuation from input query , to lower case (search engine indexes words in lower case form only), etc.)</li>
<li>
<p>however case may be needed to distinguish between acronym and real word (for example)</p>
</li>
<li>lematization: remove inflection (am,are,is<del><span style="text-align:right;">be ; car cars</span></del>> car)</li>
</ul>
<p>for reduction to lemma we need Morphology:= smallest unit that make up words (stem, affixes)
stemming := removal of affixes</p>
<p>Porter algorithm for stemming of English words</p>
<p>Step 1: cats<del><span style="text-align:right;">cat (suffix, lots of rules) Step2) walking</span></del>>walk (lots of rules like only apply if vowel before -ing); etc. etc. (well it gets rather hairy, with lots and lots of rules)</p>
<p>(2 - 5 - Sentence Segmentation)</p>
<p>How to divide text into sentences:
good markers for end of sentence are ! ?
however . - is ambiguous.</p>
<p>Need to decide if . is end of sentence - classifier: decision tree.</p>
<blockquote>
<code>
lots of blank lines after?
/ \
no-eof-s final punctuation is ? or !
\
final punctuation period?
/ \
is final word an abbrev. is-eof-s
/ \
no-eof-s ???
</code>
</blockquote>
<ul>
<li>
<p>can use probability: is this word likely to appear at end of sentence? or this this likely to be an abbreviation?</p>
</li>
<li>problem: simple decision trees with few rules are hand coded;</li>
<li>complicated trees with numeric thresholds for probability - that’s where machine learning comes in.</li>
</ul>
<p>(3 - 1 - Defining Minimum Edit Distance)</p>
<ul>
<li>
<p>measure similarity of strings = minimum number or edit operations (character insertion,deletion,substitution) to transform string A into string B</p>
</li>
<li>
<p>Levenstein distance: insertions + deletions cost 1; substitutions cost 2.</p>
</li>
</ul>
<p>string [a1……an] and string [b1….bn] (two sequences of characters)</p>
<p>D[i,j] = cost of transformaing string [a1….ai] into string [b1….bj]</p>
<p>if a1…ai and b1….bj are equal except for one character than</p>
<p>Recurrency - compute edit distance from edit distance of shorter strings.</p>
<blockquote>
<code>
D[i,j] = MAX( D[i-1,j] + 1 # inserting character to a (a becomes longer to match b)
D[i,j-1] + 1 # deleting character from a (becomes shorter)
D[i-1,j-1] + 2 (if a[i] != a[j] # substitution costs 2
D[i-1,j-1] + 0 (if a[i] == a[j]) # nothing to do.
)
</code>
</blockquote>
<p>D[I,0] = i # must delete all characters to transform a1…aj to null string</p>
<p>D[0,j] = j # must insert all characters to transform null string to a1…aj</p>
<p>Now table can be filled, with complexity O (i * j)</p>
<p>(3 - 3 - Backtrace for Computing Alignments)</p>
<p>alignment: did the character in target string come from the source string?
for each D[i,j] remember previous actions that lead to it + applied change. (if it came from D[j-1,j-1] - because last char was a fit or not)</p>
<p>(3 - 4 - Weighted Minimum Edit Distance)</p>
<ul>
<li>
<p>assign different weights to different substitutions. Why?</p>
</li>
<li>
<p>some spelling errors are more likely than others (like confusing vowels); in biology some transofmrations of DNA are more likely than others</p>
</li>
</ul>
<blockquote>
<code>
D[i,j] = MAX( D[i-1,j] + InsertionCost( xi ) # function returns cost of insertion for character xi
` D[i,j-1] + DeletionCost( yj ) # function returns cost of deletion for character xi
D[i-1,j-1] + SubstitionCost( xi, yj ) # function returns cost of substituting character xi with xj
D[i-1,j-1] + 0 (if a[i] == a[j]) # nothing to do.
)
D[I,0] = D[i-1,0] + DeletionCost(xi)
D[0,j] = D[0,j-1] + InsertionCost(yj)
</code>
</blockquote>
<p>Richard Bellman: says that name ‘Dynamic programming’ came as a marketing trick, as then secretary of defense did not like math research.</p>
<p>m(3-5-Minimum Edit Distance in Computational Biology)</p>
<ul>
<li>compare geneses from different species - to find important regions/identify functions/see how things changed.</li>
</ul>
<p>Terminology in biology: editing distance <> sequence similarity; weights <> scores</p>
<ul>
<li>in biology its OK to have unlimited gaps at start and end of a sequence.</li>
</ul>
<p>(4 - 1 - Introduction to N-grams)</p>
<p>Language model - assign probability to a sentence (one application: score sentences that come out of machine translation (???); spelling correction, speech recognition - find the more likely
interpretation).</p>
<p>p(W) = P (w1 … wn) - the probability of sequence of words.
P (W5|w1…w4) - probability that w5 follows w1…w4.
(aka. probability of upcoming word)</p>
<p>Language model - computes both probability of sequence of words + probability of upcoming word</p>
<table>
<tbody>
<tr>
<td>P (A</td>
<td>b)= P (A n B) / P (B) ;</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>(i always understood it in terms of the classical definition of probability P (A) =</td>
<td>A</td>
<td>/</td>
<td>all</td>
<td>; here this means P (A</td>
<td>B) =</td>
<td>A n B</td>
<td>/</td>
<td>B</td>
<td>as</td>
<td>all</td>
<td>is canceled out; ( set picture + probability of a given b = frequency of both a and b divided by frequency of B); don’t know why this works for all probability measures)</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>P (A n B) = P (A</td>
<td>B) P (B)</td>
</tr>
</tbody>
</table>
<p>chaining rule:</p>
<table>
<tbody>
<tr>
<td>P (An n … A1) = P (An</td>
<td>An-1 n …. A1) P (An-1 n …. A1)</td>
<td> </td>
<td> </td>
</tr>
<tr>
<td>P (An n … A1) = P (An</td>
<td>An-1 n …. A1) P (An-1</td>
<td>An-2…A1) P (An-3</td>
<td>An-4 n …. A1) …P (A1)</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>How to compute P (An</td>
<td>An-1 n …. A1) for a long sentence?</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>Markov property: P (An</td>
<td>An-1 n …. A1) ~ P (An</td>
<td>An-1 n …. An-m) (a shorter sentence will be a good approximation)</td>
</tr>
</tbody>
</table>
<p>Markov property says that a process is ‘memory less’ - it depends on the preceding state, but can do with the recent state, as long past events have diminishing effect/impact.</p>
<p>Unigram model = just look at previous word (assume Markov property of m=1) : P (A1 n … An) = P (A1) … P (An) ;</p>
<p>If this model is used to create random sentences: not very good it assumes that words appear independently of each other.</p>
<table>
<tbody>
<tr>
<td>Bigram model: P (A1 n …. An) = Product of P (An</td>
<td>An-1) over n</td>
</tr>
</tbody>
</table>
<p>If this model is used to create random sentences: does much better than unigram modell.</p>
<p>Still n-gram models are insufficient because of long distance dependencies between subject and object; for some application local context is good enough.</p>
<p>(4 - 2 - Estimating N-gram Probabilities)</p>
<ul>
<li>count frequency of each bi-grams (two word sentence) in huge corpus (text collection) in order to get probability of bi-brams. (also need count of single words - unigrams)</li>
</ul>
<p>P (W1 n W2) = countOf(W1 W2) / count(W2)</p>
<ul>
<li>column and rows are words of matrix; it holds count of occurrences for each bigram. sparse matrix
(lots of possible bi-grams never occur)</li>
</ul>
<blockquote>
<code>
P(<s>I want money</s>) = P(<s> I) P(I want) P(want money) P(money </s>) | <s>I - I at the start
of a sentence ; money</s> # money at the end of a sentence.
</code>
</blockquote>
<ul>
<li>
<p>some bigrams occur/do not occur because of grammar (allows/disallows); some because of peculiarities of the studied corpus (text set).</p>
</li>
<li>
<p>everything added as logarithm ( p1 * p2 .. pn = log(p1) + log(p2) … + log(pn) (avoids underflow and is faster)</p>
</li>
<li>
<p>toolkits for this: SLIRM ; google n-grams (even has four-grams); google book n-gram (corpus from google books)</p>
</li>
</ul>
<p>(4 - 3 - Evaluation and Perplexity)</p>
<ul>
<li>
<p>measure the accuracy against text data that is not in the training corpus (the text that was used to compute the n-gram probabilities)</p>
</li>
<li>
<p>model can be used in task (such as spelling correction) then see how many errors. (aka extrinsic (in-vivo) evaluation)</p>
</li>
</ul>
<p>intrinsic evaluation<br />
perplexity - how well can we predict the next word? (aka shannon game)</p>
<ul>
<li>perplexity PP (W) = ( p(w1,w2,….. wn) ) ^ (1/n) (normalized (^(1/n)) of words)</li>
</ul>
<p>The lower the perplexity - the better the model.</p>
<ul>
<li>
<p>as the model for n-grams is a product, then perplexity is just the mean probability of each bi-gram</p>
</li>
<li>
<p>another description of perlexity: branching factor; at each word position, how many choices are there?</p>
</li>
</ul>
<p>(4 - 4 - Generalization and Zeros)</p>
<ul>
<li>
<p>Shannon visualization method: generate random sentences from the n-gram model.</p>
</li>
<li>
<p>if you take Shakespeare corpus and wall street journal corpus for this generation: no overlap in generated sentences.</p>
</li>
<li>
<p>(for Shakespeare corpus 99.6 of all possible bi-grams did not occur) -> n-gram model only works good for text that is similar to the training corpus.</p>
</li>
<li>
<p>problem if bi-gram did not occur in test set then probability of it is 0.</p>
</li>
<li>
<p>want to assign some probability to non occurred bi-grams: La-place smoothing - add one to all occurrence counts.</p>
</li>
</ul>
<table>
<tbody>
<tr>
<td>P (wi</td>
<td>wi-1) = (count(wi wi-1) + 1) / (count(wi-1) + V) (V - vocabulary size)</td>
</tr>
</tbody>
</table>
<ul>
<li>
<p>now if we get the counts for changed probabilities - it changes them significantly; add one smoothing is not used for n-grams.</p>
</li>
<li>
<p>-> smoothing is ok for text categorization, but not for modeling.</p>
</li>
</ul>
<p>(4 - 6 - Interpolation)</p>
<ul>
<li>
<p>backoff: example there was a tri-gram with count 1 (so no much confidence in the data) -> so use bi-gram instead.</p>
</li>
<li>interpolation: mix unigram, bi-gram and tri-gram - so composite of the data is used all the time.</li>
<li>Linear interpolation</li>
</ul>
<table>
<tbody>
<tr>
<td>P (wn</td>
<td>wn-1 wn-2) = Lambda1 P (wn</td>
<td>wn-1 wn-2) + Lambda2 P (wn</td>
<td>wn-1)+ Lambda3 P (wn)</td>
</tr>
</tbody>
</table>
<p>add weight1 * tri-gram probability + weight2 * bi-gram probability + weight3 * unigram probability; sum of weights has to be 1 (to remain a probability measure)</p>
<ul>
<li>
<p>weights conditional on context: weight is not constant, it is a function on the previous word (can be ‘trained’ better)</p>
</li>
<li>now there is</li>
<li>training data: - corpus for getting n-grams</li>
<li>held-out data - corpus used to tune the weights</li>
<li>test data - used for check</li>
<li>
<p>search for weights that maximise the log probabilities for held-out data set (?)</p>
</li>
<li>what if word is not in corpus? have sepcialtoken <Unk> to mach all unknown words</Unk></li>
<li>from corpus take out a fiew rare words and reassign their probability to Unk; then get the n-grams probabilities (train the n-grams)</li>
</ul>
<p>-very large corpus (like google n-gram) what to do?</p>
<ul>
<li>
<p>one possibility is pruning (only store n-grams with count > threshold)</p>
</li>
<li>
<p>or omit based on entropy measures (more sophisticated then count only)</p>
</li>
<li>other tricks: use ‘smart’ data structures (tries); do not store strings, store index numbers of strings; use huffman coding;</li>
<li>Backoff (use of bi-gram instead of three-gram) for web scale: Stupid backoff (does not produce probabilites !)
W (n)(i-k+1) - sequence of n words starting ending at position i in the text</li>
</ul>
<p>// if both current ngram and shorter ngram (shorter by one word) appear -> take the ngram
if (count(W (n)(i-k+1)) / count( W (n-1)(i-k+1) ) then return count( W (n)(i-k+1) )
// else take count of the shorter n-gram with factor 0.4
else return 0.4 * count( W (n-1)(i-k+1) )</p>
<p>Count( W (i)(i-k+1) ) / count( w(i-1)(i-k+1) > 9</p>
<ul>
<li>recent research: discriminative models: choose n-gram modell to improve a given task (not based on corpus?)</li>
<li>caching modell: a recently used word is likely to appear again; so use probability measure that factors in recent usage. (doesn’t work good for speech recognition)</li>
</ul>
<p>(4 - 7 - Good Turing smoothing)</p>
<p>what is the probability of an unseen/unknown word?</p>
<ul>
<li>
<p>count N1 = | n-grams that occurred once | (class c = 1)
N2 = | n-grams that occurred twice | (class c = 2)
…..</p>
</li>
<li>probability that next word is unknown is equal to probability of n-gram that occured once;</li>
<li>
<p>but we need to adjust the probabilities of the remaining classes ! (otherwise it does not sum up to one - and it is no longer a probability measure).</p>
</li>
<li>
<p>modified count NewNC: NewNc = ((c+1) N (c+1)) / Nc</p>
</li>
<li>? did not quite understand how this works out ?</li>
</ul>
<p>(4 - 8 Kneser-Ney smoothing)</p>
<ul>
<li>
<p>in practice Good Turing is supposed to change most occurance counts by some fixed amount (- 0.75 ? )</p>
</li>
<li>
<p>so apply the idea of a ‘discount’</p>
</li>
</ul>
<table>
<tbody>
<tr>
<td>P-abs-discount( wi</td>
<td>wi-1 ) = ( count( wi-1 wi ) - d ) / count(wi-1) ) + Lambda(wi-1) * P (W)</td>
</tr>
</tbody>
</table>
<p>P (w) - (unigram probability)
Lambda(wi-1) - weight function takes previous word as input.</p>
<ul>
<li>
<p>problem: unigram probability is without context, so does not say much; high probability word that can appear out of contest is best fit.</p>
</li>
<li>
<p>Kneser-Ney smoothing: something better instead of unigram prob.</p>
</li>
<li>
<table>
<tbody>
<tr>
<td>should be proportional to number of different n-grams that end with a given word w =</td>
<td>{ wi-1 : c(wi-1,w) > 0 }</td>
</tr>
</tbody>
</table>
</li>
<li>
<table>
<tbody>
<tr>
<td>number of all different bigram types -</td>
<td>{ wi-1,wi : c(wi-1,wi) > 0 }</td>
</tr>
</tbody>
</table>
</li>
</ul>
<table>
<tbody>
<tr>
<td>use P-continuation(w) :=</td>
<td>{ wi-1 : c(wi-1,w) > 0 }</td>
<td>/</td>
<td>{ wi-1,wi : c(wi-1,wi) > 0 }</td>
</tr>
</tbody>
</table>
<ul>
<li>Kneser-Ney looks like:</li>
</ul>
<table>
<tbody>
<tr>
<td>P-abs-discount( wi</td>
<td>wi-1 ) = Max( count( wi-1 wi ) - d, 0 ) / count(wi-1) ) + Lambda(wi-1) * P-continuation(W)</td>
</tr>
</tbody>
</table>
<p>Lambda(wi-1) - uses constant d to cancel out d discount and reassign probability to P-continuation factor.</p>
<table>
<tbody>
<tr>
<td>Lambda(wi-1) := (d / count(wi-1)) * count( { w : count(wi-1,w) > 0</td>
<td>)</td>
</tr>
</tbody>
</table>
<p>(d / count(wi-1)) - second term when you open up the fracture expression
count( { w : count(wi-1,w) > 0 |) - number of bi-grams that start with wi-1 == how many times was d discount applied</p>
<ul>
<li>full recursive form (for all n-gram types) is very involved …. (TBD)</li>
</ul>
<p>(5 - 1 The spelling correction task)</p>
<ul>
<li>types of errors:
non word errors
real word errors</li>
<li>typographical errors (misplaced a letter and real word turns up)</li>
<li>
<p>homophones - cognitive errors)</p>
</li>
<li>
<p>fixing non word errors: find candidate fixes by dictionary lookup, pick up the words with shortest edit distance or highest noisy channel probability (see further down)</p>
</li>
<li>fixing real word errors:
candidate set = find words with similar pronunciation and spelling and also include the input word in this set.<br />
pick suggestion - noisy channel model/classifier.</li>
</ul>
<p>(5 - 2 Noisy channel model)</p>
<ul>
<li>
<p>noisy channel - input runs through a distorting communication channel; builds a modell of the distortions from the output errors.</p>
</li>
<li>
<p>decoder: runs input words through the noisy channel model - if output distortion is nearest to given observed signal (from the output end of the channel) then given input word is the match.</p>
</li>
<li>
<p>Noisy channel: given misspelled word x find correct word w</p>
</li>
</ul>
<table>
<tbody>
<tr>
<td>argmax P( w</td>
<td>v ) # find word w that maximizes probability of word given misspelling (max over all words of vocabulary)</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>argmax (P( x</td>
<td>w ) P (w)) / P (x) = # use bayes to express in terms of misspelling given correct word</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>armax P (x</td>
<td>w) P (w) # P (x) is constant so it (does not contribute to maximum ?), cancelled out</td>
</tr>
</tbody>
</table>
<p>P (x|w) - likelyhood
P (w) - prior</p>
<table>
<tbody>
<tr>
<td>P (x</td>
<td>w) P (w) = language model/channel model/error model</td>
</tr>
</tbody>
</table>
<p>correction candidate generation: need either small edit distance or small distance in pronunciation.
Demerau Levenstein edit distance<br />
for spell correction also has transpositions of two letters (in addition to insertion/deletion and substitution)</p>
<ul>
<li>
<p>big assumption: 80% of errors within edit distance 1 (or two) (? what about dyslexia, also why is it assumed that everybody does the same errors ?)</p>
</li>
<li>
<p>should also allow for insertion of space/hypen.</p>
</li>
</ul>
<p>= Channel model details: what are likelyhood of partial edits?
del[x,y] = countof( xy typed as x)
ins(x,y) = countof( x typed as xy)
sub(x,y) = countof( x typed as y )
trans(x,y)= countof( xy typed as yx)</p>
<p>(empirically compiled these, as usual)</p>
<p>P (x|w) - if deletion: del(wi-1,w) / count(wi-1,w)
if insertion: ins(wi-1,xi) / count(wi-1)
if substitution: sub(xi,wi) / count(wi)
if transposition: trans(wi,wi+1) / count(wi,wi+1)</p>
<ul>
<li>
<table>
<tbody>
<tr>
<td>unigram model for prior (P (w) factor in language model) can make mistakes; use ngram instead (do P (wi+1</td>
<td>w) and P (w</td>
<td>wi-1)</td>
</tr>
</tbody>
</table>
</li>
</ul>
<p>(5 - 3 Real - Word spelling)</p>
<ul>
<li>
<p>real word errors (mixing up two words that are correct, homophones or misuse); tougher to solve (about 40% of all mistakes)</p>
</li>
<li>
<p>for each world generate candidates: the word itself, homophones; words that have short edit distance to input word.</p>
</li>
<li>
<p>the right sentence could be any path (each one chosen from one candidate set); choose the most likely choice; Assumption: (again) there is one misspelled word in sentence (???)</p>
</li>
<li>
<p>use same channel modell (likelyhood of error edit) + n-gram language modell to check likelyhood of result; need probability of no error (assumption by application) (90% correct ???)</p>
</li>
</ul>
<p>(5 - 4 State of the art issues)</p>
<ul>
<li>
<p>problems addressed in real systems:</p>
</li>
<li>
<p>(UI issues) auto correction if have high confidence; user choice on less confidence; flag as errors if very low confidence.</p>
</li>
<li>
<table>
<tbody>
<tr>
<td>channel model in practice: lots of seperate/not quote correct assumptions happen when likelyhood (P (x</td>
<td>w)) is multiplied by Prior (P (w)) (all lies); so weight them.</td>
</tr>
<tr>
<td>argmax P (x</td>
<td>w) * (P (w) ^ Lambda) # Lambda is tuned (trial error).f</td>
</tr>
</tbody>
</table>
</li>
<li>
<p>pronunciation similarity is also often used (not just edit distance). (Aspell uses GNU Metaphone to transcribe words) ; have to combine these factors.</p>
</li>
<li>
<p>allow more edit primitives: ph -> f ; ent -> ant</p>
</li>
<li>
<p>other factors might be considered in channel model: position in word; surrounding letters; nearby keys on keyboard; morpheme transformations (like ph -> f ; ent -> ant)</p>
</li>
<li>combination of factors done by classifies (more explanation in next video)</li>
</ul>
<p>(6 - 1 Text classification)</p>
<ul>
<li>
<p>Tasks:</p>
</li>
<li>
<p>task of deciding if email is spam; (identify fishy attributes ! in title, fishy url …)</p>
</li>
<li>
<p>task of authorship attribution (deciding who was the author based on writing style - Bayesian methods)</p>
</li>
<li>
<p>gender identification (Female authors tend to use more pronous; male more determiners,facts (??))</p>
</li>
<li>sentiment analysis: tell if a review is positive or negative?</li>
<li>
<p>scientific articles: what are the key-words of an article? subject categories?</p>
</li>
<li>
<p>given text document d ; and categories C1…CN; assign category Ci to d.</p>
</li>
<li>
<p>can be done by hand written rules (if,then,else (many recursions)) problem: expensive.</p>
</li>
<li>general approach: supervised machine learning:
Input: set of classes c1…cn set of training documents d1…dm; Ready assignment of documents (trainingset) di->dj
Output: a classifiert that assigns di to class cj</li>
</ul>
<p>(6 - 2 Naive Bayes classifier)</p>
<ul>
<li>Bayes classifiers: approach is to look at the text as bag of words: only extract words deemed significant and omit all the rest; order of words does not matter (count of repetition does
apparently)</li>
</ul>
<p>(6 - 3 Formalizing Bay
es)</p>
<table>
<tbody>
<tr>
<td>P( c</td>
<td>d ) = (P (d</td>
<td>c) * P (c) ) / P (d)</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>P( c</td>
<td>d ) - prob. of assigning given document to given class.</td>
</tr>
</tbody>
</table>
<p>Now can do the usual</p>
<table>
<tbody>
<tr>
<td>find: for all classes c: maximumfor-all-c P (c</td>
<td>d) = max-for-all-c (P (d</td>
<td>c) * P (c) ) / P (d) =</td>
</tr>
</tbody>
</table>
<p>P (d) is constant, so that maximum without this division is also the real maximum</p>
<table>
<tbody>
<tr>
<td>= max-for-all-c (P (d</td>
<td>c) * P (c) )</td>
</tr>
</tbody>
</table>
<p>–</p>
<p>P (c) := the likelyhood that a document d belongs to class C.</p>
<table>
<tbody>
<tr>
<td>P (d</td>
<td>c) := by bag of word analysis we only look at set of words x1…x2 from d;</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>P (d</td>
<td>c) = P (x1….xn</td>
<td>c)</td>
</tr>
</tbody>
</table>
<p>also assume that probabilities for x1..xn are independent (not correlated) (another oversimplificiation)</p>
<table>
<tbody>
<tr>
<td>P (x1….xn</td>
<td>c) = P (x1</td>
<td>c) * .. * P (xn</td>
<td>c)</td>
</tr>
</tbody>
</table>
<p>So:
P( c | d ) = max-for-all-c P (x1|c) * .. * P (xn|c) * P (c)</p>
<p>(6 - 4 Naive Bayes learning)</p>
<p>P (cj) = countof(Cj) / Ndoc # divide number of docs of class j by number of all docs.</p>
<table>
<tbody>
<tr>
<td>p(wi</td>
<td>cj) = count(wi, cj) / Sumof-over-c count(wi, c) # divide (count of word wi in documents of one class cj) by (count of word wi in all document classes)</td>
</tr>
</tbody>
</table>
<ul>
<li>
<table>
<tbody>
<tr>
<td>Problem: can’t use it that way; if count(wi,cj) 0 then probability P(wi</td>
<td>cj) 0; then the whole product of word probabilities turns 0; (useless).</td>
</tr>
</tbody>
</table>
</li>
<li>solution: use smoothing</li>
</ul>
<table>
<tbody>
<tr>
<td>p(wi</td>
<td>cj) = (count(wi, cj) + 1) / (Sumof-over-c (count(wi, c) + 1)) = (count(wi, cj) + 1) / (Sumof-over-c (count(wi, c)) + Vocabulary-size)</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>Generalized smoothing: p(wi</td>
<td>cj) = (count(wi, cj) + alpha-constant) / (Sumof-over-c (count(wi, c)) + alpha-constant * Vocabulary-size)</td>
</tr>
</tbody>
</table>
<ul>
<li>
<table>
<tbody>
<tr>
<td>Unknown words? add token wu; count is 0 so due to smoothing: P (wu</td>
<td>c) = 1 / (sumof-over-c (count(wi, c)) + (Vocabulary-size+1))</td>
</tr>
</tbody>
</table>
</li>
</ul>
<p>(6 - 5 Naive Bayes - relationship to language modeling)</p>
<ul>
<li>
<table>
<tbody>
<tr>
<td>P (w</td>
<td>c) = unigram probability of word over corpus of documents that belong to c</td>
</tr>
</tbody>
</table>
</li>
<li>
<table>
<tbody>
<tr>
<td>And product of P (w</td>
<td>c) is the language model for this corpus.</td>
</tr>
</tbody>
</table>
</li>
</ul>
<p>(6 - 6 Multinomail naive Bayes)</p>
<ul>
<li>shows how Naive Bayes classification works in example;</li>
</ul>
<p>(6 - 8 Text classification evaluation)</p>
<ul>
<li>
<p>Reuters data set: bunch of articles classified by topics (one article in multiples categories)</p>
</li>
<li>
<p>confusion matrix: c1…cn categories confusion[i,j] = how many document from ci where assigned (by error) to category cj;</p>
</li>
<li>
<p>confusion[i,i] - correct guesses for category i</p>
</li>
<li>
<p>Precision: fraction of documents classified correctly = confusion[i,i] / sum-over-j( confusion[i,j] )</p>
</li>
<li>
<p>Recall: fraction of documents classified as category j that were correct = confusion[i,i] / sum-over-j( confusion[i,i] )</p>
</li>
<li>
<p>accuracy: fraction of documents classified correctly = sum-over-i( confusion[i,i] ) / sum-over-i( sum-over-j ( confusion[i,j] ) )</p>
</li>
<li>
<p>micro and macro average:</p>
</li>
</ul>
<p>for each class prepare table: CLi</p>
<blockquote>
<code>
classifier-is-true classifier-is-false
classifier-says-doc-belongs-to-class-n <count-of-docs> <count-of-docs>
classifier-says-doc-not-belongs-to-class-n <count-of-docs> <count-of-docs>
</code>
</blockquote>
Microaveraging: add such matrixes CLi and compute precission over resulting table. Problem: high frequency classes have high weight.
Macroaveraging: compute precision over the table CLIi for each class and then compute average of these values.
- input data:
training set - to learn the modell;
development set - to tune parameters (during development)
test set - to evaluate the performance of the result (after tuning)
- cross validation: take multiple pairs of (training set, development set) and check; this is designed to prevent sampling errors, too few data; overfitting (too much tuning against development
set); etc.
- check each pair with the test set; aggregate the result for each triple/input data set.
(6 - 9 Practical issues in text classification)
- what approach is appropriate for a given task?
no training data: hand coded rules (tree of nested if-then-else)
little data: use naive bayes - it does not overfit on small amount of data.
semi-supervised learning: start with small amount of data to train/get larger amount of data (not covered here)
large amount of data : regularized logistic regression or support vector machines (covered later) (problem: takes a lot of time to train)
- ? if huge amount of input data then most methods converge?? (study with spell checking over huge input)
- real system often combines: automatic classification + manual review of new cases (?)
- combinee user tunable decission trees (some users like to hack them) (?)
- practical problem with bayes: lots of multiplication can result in underflow (imprecission);
therefore compute log probabilities log(x \* Y ) = log(x) + log(y) : addition better than multiplication.
- during parsing of imput:
- substitute a category of input for one token (example: all part numbers -> part-number-token)
- stemming (substitute word with stem) does not help here (?, bag of words ?)
- upweight: some parts of a document are more important; occurance of a words in title is counted twice; also true for words that appear in first sentence of paragraph; etc.
(7 - 1 what is sentiment analysis)
- task: is a product review positive or negative? (one measure)
- task with multiple measures - called aspects, score each: ?is it easy to use?, ?is it of value?, ?is customer service good?
- task in politics: sentiment analysis of twitter tweets (strongly correlated with gallop polls)
- What is sentiment?
- sentitment is an attitude (Psychology: Scherer typology of affective states)
- (enduring, affectively colored beliefs, dispositions towards objects or persons): like, love, dislike, hate; etc.
- has a source (who has the attitude) and an object (towards what); and type of attitude and strength
(7 - 2 Sentiment analysis, a baseline algorithm)
- tokenizing of words (preserve capitalization, as this also expresses sentiment; parse emoticons ;-( ) ; but phones and dates normalized.
- feature extraction (mostly words, but not only; deal with negation (didn't like the movie)
- classification
bb
- how to deal with negation'
"didn't like this movie." -> "didn't NOT\_like NOT\_this NOT\_movie." (NOT\_ appended to every word after negation till eof sentence.
- again: find category that maximizes: argmax-over-c P (cj) \* product-over-i( P (wi | c ) )
P (wi | c) = (count(w,c)*1) / (count©*|V|) \# Naive Bayes with - Laplace smothing. just like with spelling.
- for sentiment analysis - slight variation is used: Multinomial Naive Bayes. here we care less about frequence of a word; what is important that a word occurred or not.
- -> clip count to one. so for each document considered we remove duplicate words.
- Problems: subtleties/word plays that do not use any clear terms.
- Problems: thwarted expectations: 'this film should be brilliant .... but it can't hold up' (Bayes does not do that)
- Problems: (he did not mention) changing patterns of expression? Is it all that static over time?
- problem: if class1 has more reviews by order of magnitude than class2; then P (c) will skew the results in favor of c1.
solution: select no more then |class2| reviews from among class1
(7 - 3 Sentiment lexicon)
Thesauruses of sentiments:
- Example 'the general inquirer' (has categories (like 'positive', 'active' 'passive') and words that fit into categories)
- LIWC (Linguistic inquiry and word count) (>70 classes)
- MPQA subjectivity Cues lexicon
- SentiWordNet
- again can't use raw count of words; will use likelyhood: P (w|c) = count(w,c) / sum-over-w-in-C( count(w,c) ) \# number of occurrences of word in class / number of words in this class
- that makes more sense: 'great' and 'awsome' mainly used in 10 star reviews; 'terrible' in very negative reviews; 'good' does not say much.
- negation words: 'no' 'not' 'never' more likely to appear in low star reviews.
(7 - 4 learning sentiment lexicon)
Make your own lexicon by means of semi-supervised learning. Also good to extend a smaller lexicon into larger one.
- observation: adjectives joining by 'and' have same polarity ('fair' and 'legitimate') joined with 'but' the are often opposites
(7 - 4 Other sentiment task)
- finding sentiment of sentence; find the target(/aspect/attribute) of a sentiment;
'the food was great but the service was awful' - two targets!
- find highly frequent phrase: sentiment word right before such a phrase marks a sentiment 'great food' in restaurant reviews.
- if domain is fixed and known then one can analyze a set of reviews and extract the subjects manually (aka. aspect/attribute/target)
- problem: if class1 has more reviews by order of magnitude than class2; then P (c) will skew the results in favor of c1.
solution: penalize(punish?) classifier for misclassifying instances of C2.
- pipeline approach:
text -> subdivide into sentences and phrases -> run sentiment classifier (know which sentences are positive, negative) -> aspect extractor (what is the subject) -> aggregator (statistics)
- other problems: classification of other 'affective states' (Emotion (angry,joyful,etc) Mood (cheerful, sad), Interpersonal stance (friendly, hostiles) etc. )
(8 - 1 Generative vs discriminative models)
- generative models (aka joint models) : - assign probabilities & generate the observed data from hidden (= training set) data (naive bayes, n-grams)
P (c)->P (d|c1)
observe probability of class c in training set (as given) and infer probability of word given the class
try to find maximum of P (d|ci) in order to find class cj that word belongs to. (good for estimation)
- discriminative models (aka conditional models): data is given, assign probabilities to hidden (=training data=) ; are supposed to be language independent (?)(Logistic regression, maximum
entropy, SVM, Perceptrons)
P (c|d1...dn) -> P (c)
observe probability of class given words and infer probability of the class.
here we try to find maximum of P (c|d) -> maximum likelyhood of class instance given the input data. (says they are more accurate than gerative modells, but pr prone to overfitting)
(8 - 2 making features for text)
feature:= elementary piece of evidence that links data (d) with category © (reminder: want to predict likelyhoold of category c given data d)
feature: C x D -> R : feature is a function with two arguments, first one is a class instance cj, second one a word di ; result is a floating point number (return value is called weight of
a feature)
example:
f(c,d) = \[ c is class LOCATION; and d is word wi which is capitalized and wi-1 (preceding word) is in\] matches 'in Berlin' 'in Paris'
- Each feature picks out data subset (of words d) and suggests a label (class c) for it
return value (called weight of f is true false; this number is tuned by the 'model' (the 'model' uses feature functions as tools)
-positive weight indicates that feature is likely correct
-negative weight indicates that feature is likely incorrect
Expectation := actual or predicted count of instances where input data triggers/satisfies the input condition of a feature (feature 'fires')
empirical E (fi) = Sum-over-c-d fi(c,d)
empirical expectation of feature is sum of all instance where feature does apply (for given input data,category pair)
Model expectation of feature
E (fi) = sum-over-c-d P (c,d) fi(c,d) - weighted by probability of word d in category c.
In NLP a feature returns boolean (yes or no) value; so
fi(c,d) = {(d,cj) | Predicate(d) ^ cj\] Predicate(d) - returns true/false; result is boolean;
Feature based model := all decisions of the model are based upon return value of feature functions over input data.
(followed by some more example features used with text classification and text disambiguiation tasks)
- turns out that quality of feature is more important than type of statistical model (all models about the same, given same features ;-)same features)
- also used smoothing, important for discriminative models.
classifiers are used in other tasks as well: sentence boundary detection, sentiment analysis, PP attachment (grammar disambiguation); parsing decisbions.
(8 - 3 linear classifiers)
Linear classifiers:
- for each class © and input data (d) compute vote
- vote© = sum-over-i Lambda-i fi(c,d) \# each classifier fi has its weight lambda-i (can be positive or negative)
- the category that gets the maximum number of vote© wins; and d is assigned to its class c.
Mechanism for adjusting weights (Lambda values)
- perceptron: find misclassified example and change weights so that it is correctly classified.
- support vector machines (margin based) (not covered)
- exponential (aka. log-linear, maxent, logistic, gibbs) : probabilistic model
- problem vote© can be negative (probability can't); solution: use exp(vote) -> always positive.
- make it a probability: P (c|d,lambda) = exp( vote(c,d) ) / sum-over-all-ci exp( vote( ci, d ) ) \# for a given set of lambda constants.
- to write it out
P (c|d,lambda) = exp( sum-over-i Lambdai fi(c,d) ) / sum-over-all-ci ( sum-over-i Lambdai fi(c,d) ) )
b
- intuition of normalizing (turning into probability): features with strongest vote dominate more strongly
- task: find lambda's that maximize conditional likelyhood P (c|d,lambda) of data.
(8 - 4 building maximum entropy (maxent) models)
- manually: create the features and tuning of lambda factors; evaluate statistics; tune, add features to exclude bad decisions and repeat.
(8 - 5 more about advantages of maxent models (compared with naive bayes))
Bayes model: if composite words "Hong Kong" if Hong and Kong are treated as seperate words, then evidence/probability is double counted in Bayes model! This skews classification; Bayes double/multi
counts correlated evidence.
Maximum entropy solve this: the weighting feature helps to adjust the model expectation to observed expectation.
(8 - 6)
how to find the lambda coefficients?
- optimization problem: look at all possible solutions for all possible lambda values (each lambda-i is a dimension); then find maximum/optimal solution (like newton method for multiple
dimensions)h
- therefore need derivative/gradient (calculus) of
delta log(P (C|D,lambda)) / delta Lamba-i
(trick: compute that over logarithm of the exponential probability - so that exp and log will cancel each other out)
log P (C|D,lambda) = log ( Product-over-all-c-and-d P (c|d,lambda) ) \# because of independence =
= Sum-over-all-c-and-d log( P (c|d,lambda) ) \# because log(a \* b) = log a + log b
\# spelling out definition of P (c|d,lambda)
Sum-over-all-c-and-d ( log ( exp( sum-over-i Lambdai fi(c,d) ) / sum-over-all-ci ( sum-over-i Lambdai fi(c,d) ) ) ) =
\# because log(a/b) = log(a)-log(b)
sum-over-all-c-and-d log (exp ( sum-over-all-i ( lambda-i fi(c,d) ) ) ) - sum-over-all-c-and-d log ( sum-over-all-c ( exp ( sum-over-all-i lambda-i \* fi(c,d) ) )) )
= N (lambda) - M (lambda) \# N and M will be tackled independently
sigma N (Lambda) / sigma lambdai = ..... Sum-over-c-and-d( fi(c,d ) ) \# the empirical/(actual) count of fi,c
sigma M (lambda) / sigma lambdai = ..... sum-over-c-and-d( sum-over-c\`( P( c\` | d, lambda ) \* fi(c\`|d) \# predicted count
Sigma P (C|D,lambda) / sigma(lambdai) = empirical/(actual) count of (fi,c) - predicted count of (fi,lambda)
- Maximum is reached when derivative is 0; this happens when empirical count is equal to predicted count.
- so solution is unique and always exists (???)
- use gradient descent optimization (walk a bit in direction of gradient, then repeat until maximum found)
(9 - 1 Named entity recognition)
- Information extraction: produce structured representation of relevant information by extracting/processing of limited relevant parts of text.
- output: some formalized relations (like DB); knowledge base
- goal: output should be useful for humans who want to make sense out of data, or input for next stage of computer processing.
- extract factual information:
example: gather info about companies from company reports.
example: drug-gene interaction from medical literature (BIO text mining)
- Task: named entity extraction: what does name refer to? assign name to class: Person/Location/Organization
- useful as first stage of sentiment analysis: what do they talk about?
- usefule as first stage for question answering:
- web: give links to pages that refer to domain of name
(9 - 2 evaluating named entity recognition)
- evaluation per entity (Hong Kong) not per token (Hong, Kong)
- example error: First Bank of Chicago has .... -> extract 'Bank of Chicago' instead 'First Bank of Chicago' ; this is called boundary error.
- Problem: this errors is counted as two errors: 'First Bank of Chicago' is not recognized -> ** false negative; 'Bank of Chicago' -> ++false positive.
Problem: returning no match at all what have scored better !
- Solution: system where you get partial credit for partial matches like this.
(9 - 3 Sequence models for entity recognition)
- collect set of training data; in training data each token is assigned its entity class (labeled) (label as Other if no entity class matching)
- design feature extraction functions
- train sequence classifier (get the lambdas) over the training set to predict label from data
- test the classifier over real data and adjust model; repeat.
- IO encoding: label token as O for other if no class applies; label class name if class applies: problem: names that are sequence of tokens "Gerorge Washington" )
- IOB encoding: if a sequence of words belong to a class then first token of the sequence is labeled as B~~<span style="text-align:left;">classname> (B</span>~~ beginning); the reset of sequence labeled as I-<classname>; O for other)
problem: now we have two times as many tokens (+ 1 for other)
- IOB encoding solves problem like "Fred showed Sue Mangquin Huang's new painting" - but this happens rarely, Stanford library of NLP uses IO encoding)
- classifier functions consider current word, and previous and next word (for some context); Part of speech tags (POS) can also be used);
- sequence classifier also considers previous and next token label (extra conditioning) (this makes i a sequence classifier)
- also consider character subsequences (substrings) of tokens: (like 'oxa' substring to identify drug names, colon indicates movie names; field suffix -> indicator for place name)
- also to consider: word shapes = defined as follows;
substitute lower case letter for x; upper case for X; digits for d; lower letter sequwnces longer then four substituted with one x;
resulting pattern can be useful for classification (abbreviations, etc)
(9 - 4)
Many tasks on sequences of words as input:
- POS tagging (sequence of words, classify into NN,VBG,DT, etc.)
- named entity recognition (labels: Person, organization name)
- word segmentation (Chinese/Japanese characters,compound words in German/Turkish) (labels Begin or word and inside or word)
- text segmentation (find sentence boundaries, is this sentence a question or answer?)
- classify each line as question or answer
classification at sequence level: use conditional Markov Models - makes decision based on current data + some previous decisions (not too far back)
,
domain: extracts token/features: W0 current word; previous word W-1; T-1 - previous tag, parr of two previous tags T-1T-2
(? how do POS taggers do in languages with free word order ?)
- Beam inference: at each position keep several posibilities (rather than deciding the current label); so keep top k complete sequences and extend each subsequence them to k+1 length fs/decide on
on the best fit.
-Viterbi inference (another form of dynamic programming): works if you decide to examine a fixed size window of past input (say always two last tokens); can use recurrencies to find optimal state
sequence -> dynamic programming.
(says that restriction on past input is not so bad, as long distance dependencies are difficult to do)
- conditional random fields
(10 - 1 relation extraction)
- relation: predicate(argument1, argument2) example: founding-year(ibm,1911)
- relations can be more complex (like SQL tables with lots of columns/attributes)
- aim is to create structured knowledge bases, thesauruses
- example system (ACE - automatic content extraction) ; extracts tables from text like located-in; part-whole; family-relations; has meta relations PERSON-SOCIAL and instances of meta relation
(Family, business, lasting personal)
- research systems like to mine relations from wikipedia ...
- format of relations - RDF triples: subject predicate object - used in DBPedia project
- Wordnet thesaurus likes is-a and instance-of relations;
(10 - 2 hand written patterns for relation extraction)
X is-a Y relationships are indicated by certain forms:
X such as Y
X or other Y
X and other Y
Y including X
Y, especially X
that is also true for other verbs that often denote specific relations:
located-in(ORGANIZATION,LOCATION)
founded(PERSON,ORGANIZATION)
cures(DRUG,DISEASE)
also observes the following patterns:
PERSON, POSITION-IN-HIERARCHY ORGANIZATION
PERSON (pres = appointed|chose|was named) POSITION
says that these patterns are of high precision; problem: hard work to cover all possible forms.
(10 - 3 Supervised machine learning for entity extraction)
- again: training data,test data, etc. etc.
- stage1) build yes/no classifier that answers if two entities are related; all pairs are extracted.
(this one is very fast; stage1 acts as a filter)
- stage2) decide if entities returned by stage1 are related then classify relation (supposed to be slow)
designed to be fast running.
- for entity extraction we extract features( this will be the input of the matchers)
b
For stage 2) extract Relevant features (well all we know)
Headword of entities to match (like American -> Airlines : Airlines is the headword)
bag of words between features to be matched.
sequence of bigrams between the entities (or in all the sentence)
named entity assignment (output of named entity recognition), or new concatenation type of the two entities involved)
'entity level of a matched entities (can be name nominal or pronoun)
parsing feature: (consider feature of lowest level of noun phrase / verb phrase (?what about ambiguous parses?)\]
can consider higher order relationship within parse tree.
consider dependency path
Gazeteer feature (
any long list of proper nouns like
- list of common names;
- list of geographical
- locations like country name list; etc)
- Evaluation of results:
Precision := \# of correctly extracted relations / \# number of total extracted relations
Recall : = \# of correctly extracted relations / \# of gold relations
F1 := (2 \* Precision \* Recall / (Precision + Recall)
- can get high accuracies
- problems: high cost of hand labeling training sets; brittleness: don't generalize well to unseen data.
(10 - 4 unsupervised learning)
- small training set, or few seed patterns; the algorithm extends it to a larger set of patterns (bootstrap)
Hearst 1992:
- given relation R between seed pairs (example: Mark twain buried in Elmira, NY)
- find sentences with the given relation R; (search for "Mark Twain' and 'Elmira')
- for search results: look at adjacent text around the found instances of relation R; try to generalize and find new patters.
(result pattern: the grave of X is in Y; X is buried in Y)
- repeat with the new patterns added
Brin 1998: was looking for author book pairs:
- Search of 'The comedy of errors' and 'Shakespeare'
- look at matches: x?, by ?y x?, one of y?'s (looks at words between and context after - if several matches of the same form, extract pattern)
Snowball algorithm:
- add requirement: x and y must be named entities;
- also computes confidence value
Distant supervision: (?)
- instead of seeds take DB; of relation
- for each relation in DB (take Born-in); take known facts from DB; (Einstein born in Ulm);
- search for Einstein and Ulm;
- from hits: extracts lots of features
- features: input for supervised classification (positive instances taken from DB)
- find conditional probability of relation given the features.;
Textrunner: (extract relation from web with no training data)
- train classifier to decide if tuple is trustworthy.
- in corpus: take adjacent simple noun phrases (NP) and keep them if previous classifier says that they are trustworthy
- rank these relations by redundancies (often used patterns are more likely)
Evaluation of semi supervised algorithms?
- can't do the usual; don't now what was missed and how many are there.
- draw random sample and check precision ,recall, etc from this ouput.
- but no way to measure the system as a whole (just sample output)
(11 - 1 Maximum entropy)
------------------------------------------------------------------------
(15 - 1 CFG - context free grammars and PCFG (P for probabilistic))
Context free grammar: rewrite rule that do not look at any context (just the rules)
S -> NP VP
VP -> V NP
VP -> V NP PP
- Computer science CFG definition:
G = (T, N, S, R)
T - terminal symbols (tokens, lexicon words)
N - non terminal symbols (nodes within parse tree)
S - start symbol (is a non terminal)
R - rewrite/production rules
- The same for the purpose of NLP:
G = (T, C, N, S, L, R)
C - preterminal symbols; tokens grouped together as one class like DT -> the, NN -> man, women ...
Lexicon; now looks as rewrite rule Pre-terminal -> terminal (assigns terminal to its pre-terminal category)
Rewrite rules:= left hand side is non terminal; right hand side consists of pre-terminals or non terminals
Sometimes they have ROOT or TOP symbol as the real start symbol (instead of S)
- PCFG - probabilities added
G = (T, C, N, S, L, R, P)
Adds probability function P that assigns a probability of applying each particular rewriting rule.
P: R -> \[0..1\]
For each non terminal the sum of probabilities for the rewrite rules where this non terminal is left hand side is 1.
NP -> NP NP probability 0.1
NP -> NP PP probability 0.2
NP -> N probability 0.7 (sum is one)
the tokens generate unigram language model (sum of probability of all tokens is one)
(of course the sum of the probabilities for each rules is 1, that makes it a language model)
Def: probability of a parse tree is the product of the probabilities of each production rule. (intuition: product of probabilities is probability; also deeper parse trees get lower probability
because product of numbers <1 gets smaller and smaller)
the assumption is to choose the parse tree with the biggest probability.
Def: probability of a sentence: sum of the probabilities of each possible parse tree of the sentence
(15 - 2 grammar transformation)
Chomsky normal form := all production rules have one or two terms on the right hand side; (reason for this form is that it is more efficient to parse productions like this)
Can transform any CFG into Chomsky normal form (so that parsing is easier)
How?
If grammar has rule that always applies
NP -> (empty)
look up rules where N is used at like
S -> NP VP
and split it into two rules
S -> NP VP
S -> VP
Remove unary rules of the form
S -> VP
go to definition of VP rule and add new rules that transform S to right hand side of VP rules.
rules with more than two terms in right hand side are also split
VP -> V NP PP
into
VP -> V NP@
NP@ -> NP PP
- when parse tree is compared with parse tree from Penn Treebank: reduce treebank categories to parser categories; remove tree nodes with branches that reduce to empty.
(15 - 3 CKY Parsing)
Please note
![](/cstuff/images/nlp-cky1.png)
- first apply all unary rules to the words; cells in diagonal of the parsing triangle are filled in. (keep backtraces to get the result (like with edit distances)
- then fill the inner nodes: reduce input from bottom up and from left to right( node(1,0) node(2,0) ,
- for inner nodes upward: try to combine existing parse trees of given length; for word1...word3 -> try tree(word1) tree(word2..3) and tree(word1..2) with tree(word3)r
for node(2,2) then this one can be parsed by combining node(0,0) and node(2,1) (cor by combining node(1,0) and node(2,2) - difference in choosing the 'split point' along the x axis)
if two productions from the same left hand side are possible: chooses the result with highest probability:
(15-5 evaluating parsers)
- Label precision: compare with parses in treebank; each node (constituency) and its positions are recorded (NP (2,4) VP (5,7); compare number of coinciding nodes between tree bank parse and parser
output. as usual computer precision/recall.
- for wrong noun phrase attachment the result will be low (cascading errors through the nested tree)
- P (probabilistic) CFG: result will be 73% for F1 measure; not great; at least it tries to deal somehow with grammar ambiguity.
- very bad as language model: because of being context free (likelihood of rewrite rules is not conditioned on words that make up the production); (even n-grams give better results)
(16-1 lexicalized PCFG's)
- lexicalized CFG: for each category name (VP,NP) add the head word of the current clause as it appear in parse tree to category name;
Define head word for NP: (NP-<last noun="" in="" NP="">)
head of VP : (VP-<verb that="" appears="" in="" verb="" phrase="">)
S : (S-<verb that="" appears="" in="" verb="" phrase="">)
PP : (PP-<preposition that="" appears="" in="" PP="">)
Problem: number of non terminal explodes ! Advantage: can condition probabilities of a parse on its head words: can condition PP attachment for particular prepositions/head word combinations. (does
PP attach to VP or NP?)
(16 - 2 Charniak model for PCFG's (1997))
- parsing from bottom up (like CKY)
- probabilities condition from top to bottom (need to know probability of parent node prior to finding probability for node)
- P (h | ph , c, pc) : probability of head word h word of a parse node is h : condition given the probabilities of ph - parent head node's head word; c - category of the node ; pc - parent category
of the node)
- P (r | h, c, pc) : probability of production rule : conditioned on given probabilities of head word (h); category ©; parent category (pc)
(S (NP VP))
S - has pc (parent cateogy pc - S) ; ph (parent headword)
NP - - has h - headword; c - category (NP); r - production rule chosen to expand NP
- the aim: to know the probability of expanding a given word with a given rules.
(? does that scale to large corpus ?)
- problem: hard to predict likelihood for probability distributions: things get very sparse;
- solution: (like with language models) interpolation between different models
P (h|ph,c,pc) = Lambda-1(e) P-MLE(h|ph,cpc) +
Lambda-2(e) P-MLE(h,C (ph),c,pc) +
Lambda-3(e) P-MLE(h|c,pc) +
Lambda-4(e) P-MLE(h|c)
(what a mess, probably only good for small corpus/tree bank)
(16 - 3 independence assumption of PCFG's)
- independence assumption in pure PCFG's - no assumption on context; only looks at label of the node; content of node is of no interest to the model.
- that's to tall: NP under S or NP under VP: likelihood of productions for each of these cases is very different.
- als PCFG's do not perform too well (often choose the wrong parse as the most likely one)
- really need context to disambiguate posessive NP ; ( ( NP (NP (Fidelity) POS ('s)) II (new) NN (toy))
((NP (NNP Big) (NNP Board)) (JJ composite) (NN trading)) <-> (NN (NNP Big) (NNP Board) (JJ composite) (NN trading))
- what helps is state splitting (Johnson 98) to non terminal name - add name of parent non terminal ((S-ROOT) (NN-S) (VP-S))
- problem: too much state splitting leads to sparseness of probabilities (bad) need smoothing, etc
(16 - 4 Unlexicalized PCFG's)
\[Klein Manning 1993\]
- does not add head word to non terminal
- allow more states (like NP-S but do not add all of them wholesale); or add other features NN-CC (CC - coordinative noun phrase)
- target: make limited number of state splits (to keep grammar small and fast (and prevent smoothing))
- assumption: most of the state splitting consider/take into account grammatical features: finiteness, presence of verb auxiliary.
- other work: (Charniak Collins):
Problem: rules like NP = NNP NNP NNP; these rules are split to binarization; this creates deep nesting structures so
that probabilities get split so that they get very small (they are multiplying probabilities in PCFG);
Solution: only consider 1 or two adjacent expansion (Markovization - consider only limited context;see 'memory assumption of markov models')
Horizontal markovization: for long rules consider only probability first of first two non terminals
Vertical markovization: also look at probability of parent category (or also parent of parent))
- other problems: unary rules (like S : VP ) are used to swap categories, so that a high probability rule can then be used on the sub tree for the right hand side of the unary rule)
problem that these unary rules do not apply for specific context
solution: add context that say when to apply unary rule (by state splitting S, S-U where to allow uniary rewrite)
- other problems: tree bank cateogories (non terminals) are to coarse (example: tree bank IN is used for sentence complementizer (that weather, if) ; subordinating conjunctions (while after) and
true prepositions; fix is to subdivide the IN category.
- PP attachment problem: want to know if PP attaches low or high in the phrase; so that can be conditioned on the verb that appears with the NP
(16 - 5 latent variable PCFG)
- \[Petrov and Klein 2006\] tries to do state splitting by machine learning (unlexicalized categories)
- for each category: tries to split categories if performance is improved, keep it else roll back. (does not go into the details)
- resulting splits reflect semantic categories: (proper nouns, first names, second names, place names) same for pronouns
(17 - 1 Dependency parsing)
- Head word/governor is the node;
- node points to its dependents
Result is a tree (that means no cycles and one head)
by convention they have a ROOT node that points to head of the sentence
= how does that relate to context free grammars? Nowadays categories are annotated with the head word (lexicalized CFG's; x-bar theory); so one assumes that there is dependency between nested
consistencies then CFG is a kind of dependency grammar.
- difference: dependency grammars have more flat parse tree:
CFG: (S-walked (NP-sue) (VP-walked (VBD-walked) (PP-into ))) ;
in dependency grammar walked is dependent directly on on sue; walked; into.
- in dependency grammar you can point across spans (crossing dependencies); however CFG is strictly subtree, can't do that.
- dependency parsing: one possibility is dynamic programming with CKY; or maximum spanning tree algorithm
- preferred: left to right parsing (! MaltParser does O (1) !)
- what are the features available ?:
Valency of heads: how many dependency are usual for a head? are Dependencies attached to the right or left of the word?
dependency distance (mostly within nearby words ?) (? true for languages with free word order ?)
bilexical affinities: pair or words that usually appear together (the <-> issues)
intervening material: dependencies rarely cross verbs or punctuations (commas they do cross)
(17 - 2 Greedy transition based parser (example: MaltParser))
- makes local (greedy decision)
- left to right 'shift' 'reduce' parser; has a stack
- Shift: take next word in input and push to stack
- Reduce: take next input word and a term on the stack (DG: the dependents/CFG: the left hand side of a production) and combine them/reduce it to one symbol; put result on the stack
(DG: the head word/CFG: the right hand side of production)
Left arc reduce: (adds <- arrow) take word to the right as head
Right arc reduce: (adds -> arrow) take word to the left as head
(also need to attach label on to the resulting arc)
Problem with this approach: if all the arcs point to the right then have to shift all words until the last one, then reduce reduce reduce; (Lots of shift means large lookahead: hard to tackle with
machine learning techniques).
- Solution: arc-eager parsing:
Idea: allow to establish right arch dependencies before they are fully resolved
- Left arc reduce (builds <~~) again between \[input word\]<~~\[head of stack\] ; only if \[head of stack\] is not already head of dependency (rule to prevent situation that stack is pointed to by two
arrows)
- Right arch reduce (builds ~~<span style="text-align:right;">) reduce \[input word\]</span>~~> \[head of stack\] ; also take \[input word\] and \[push it to stack\] - so that it will find the head word of this clause !
- extra reduce: once the head word has been found; reduce this word of the stack.
- to decide if to reduce and direction of arch: train classifier with maxent model that decides on the local step. Available features: top of stacks, POS, first in buffer, its POS; length of arc,
words between arcs. etc.
- O (1) - only greedy/local decisions made at each step ! and that's almost as good as the best LPCFG and its much faster.
Evaluation of parser performance: each word in sentence is numbered : 0 - Root of sentence; 1 - first word; for each word: what is the index of its head? what is the function of the word?
as usual: Accuracy = \#correct dependencies / \# of dependencies;
Problem in dependency parser: crossing dependencies
who did bill buy the coffee from yesterday
buy: head of ,did ,bill, coffee, from, yesterday.
but who is the heda word of from ! this is crossing dependency that spans across the tree (can't have that with CFG)
Now arc eager parsing can't do that (area of research)
(17 - 3)
one of the advantages of dependency trees is that they map naturally into a semantic representation (CFG would require additional 'deep structure' transformations over the parse tree, just to make
sense of the structure !)
- example: task of extracting interactions between genes/proteins from biomedical literature:
picture dep-1
the verb interacts indicate relations; dependencies indicate subjects and objects of interactions. (see conjunction that indicates two objects)
- stanford dependency representations: represents dependencies as projective dependency tree (can get it from post processing of penn tree-bank)
(jumped (boy-nsubj (the-det) (little-amod)) (over-pre (the-pobj (fence-det))))
boy-nsubj : nsubj - the label of the arch between jumped and boy
- other changes made for relation extraction:
- collapsing nodes (based (in (LA)) -> (based (LA-prepIn))
conjunction: both makes and distributes point to subject (now it's a graph and not a tree)
![](/cstuff/images/dep-2.png)
- looking at results of BioNLP (competiton of relation extraction in bio):
- Most relations are 1-3 dependencies away; but linear distance between the subject and object is up to 5-6 words;
- this means that dependency representation is very efficient (shallow tree is easy to inspect for semantics step)
(18 - 1 information retrieval)
- works on large collection of (usually) unstructured text (set of documents)
- task is to satisfy a information need and to find relevant information from this heap.
Workflow: user has a task; formulates an information need; then formulates a search query; query is passed to search engine, based on results the user may have to revise previous staps
Evaluating search results:
Precision := number of relevant documents returned by query / number of results returned by query
Recall := number of relevant documents returned by query / number of existing relevant documents
(Problem - these are subjective with respect to users information need, these are low if query is not well formulated)
(18 - 2 term document matrix)
- rows of matrix: all words in the corpus (lower case, after stemming to keep count low)
- columns: all available documents
- matrix\[i,j\] = 1 if word i occurs in document j
- very sparse, so it is impractical to store such a structure
(18 - 3 inverted index)
- each document identified by document-id (serial number)
- for each term in corpus (lower case after stemming): keep list of document-id's where term occurs in. (sorted by document id)
- term1 and term2 := because list is sorted by document-id: merge list and return documents that occur in both lists
which terms are indexed (have a list of document-id's where they occur?)
- text is tokenized (John's -> John, state-of-the-art (remove hyphens); remove punctuations.
- normalization: U.S.A -> USA; stemming
- olden times: remove stop words (the, a, to, of) ; now they don't do that because it kills phrase searches
Processing:
after that sort terms of each document: then produce pairs (term, document-id); then sort-merge all pairs of (term,document-id) (stable sort so that documents are lined up by document id)
- they try to compress the indixes, because they take up a lot of space.
(18 - 4 query operations)
- term1 and term2 := because list is sorted by document-id: merge list and return documents that occur in both lists
(18 - 5 phrase queries)
- possible solution: add bi-word index (each bi-gram as entry in index); then each for (query-word1 and queryword-2) and merge with (query-word-2 and query\_word-3)
- problem: not quite exact: can have false positives for words do not appear in adjacent positions.
- problem: index much more massive (not good for tri-grams)
- Positional index: need to extend inverted index: in addition to document id need the position of match in document pair : document-id:position-in-document
- these also need to be sorted by position (doc-id-1:1, doc-id-1:10, doc-id-200:5, doc-id-200:55, doc-id-200:402 )
- for matching document: merge the position list in the same way to find adjacent matches.
- can also do proximity queries: term1 within n words of term 2
- problem: positional index is larger (especially with long documents) than simple inverted index. Positional index 2-4 times larger; and is now about 35-50% of original text size !!!
- combined approach: add common bi-gram's (appear frequently as search terms) to index ; claims it uses 26% more space but 1-4 of the time to process)
(19-1 ranked retrieval)
- boolean search: difficult for users to handle; return too many (typical for or query) or to few results (typical for and queries)
- need more relevant results to appear on top/at the beginning of search result list
- assign score to documents: 0..1
- (altavista) for single word queries: 0 - word does not appear in document; >0 word appears in document; higher result if word appears more frequently (or in the beginning or in the
title)
(19-2 Jacquard coefficient)
jacard(A,B) = | A n B | / | A u B | (number of elements in intersection divided by number of elements in union)
0 - for disjoint sets ; 1 if A eq B ;
Problems:
- does not consider term frequency;
- also search terms with rare frequency (\# of occurrences of term) are more informative then terms with large frequency (jacquard does not take this)
slightly better measure:
jacard(A,B) = | A n B | / SQRT( | A u B | ) \# size of document should not skew results by much
(19 - 3 Term frequency weighting (TFW))
- assume we know how many times a term occurs in document
(do not consider order of words - bag of words model - this has its limitations, as positional info is not considered)
- assumption: documents where search term occurs a lot is preferred over documents where search term occurs just once (however measure should not grow linearly with occurrence count)
- treat words in document as bag of words (order does not matter) (? this throws away useful information: does not assign higher weight for words that appear in title or first passage?)
also term frequency Tfd
count-of-term-in-document
- take 1 + log( count-of-term-in-docubment ) \# if count-of-term-in-document > 0 (log - base 10.)
0 \# count-of-term-in-document == 0 (because log(0) = -infinity
sum that over all search terms in the query.
document sore of document that matches search query
(sum-over-term-in-query-t ( 1 + log-10 ( count-of-term-in-document-that-matches-sarch )
(19 - 4 term frequency/document frequency weighting (TF-IDF weighting)
- assumption: rare words more informative than frequent words; give high score for rare words
document=frequency( term )
number of documents where term occurs in \# the more this count, the less the information value of a term
- want inverse of document frequency (IDF): the less the count the higher the score
inverse-document-frequency( term )
log-10 ( Number-of-documents / document-frequency(term) ) \# log to dampen the effect
- score( query-result) = sum-over-search-terms-that-appear-in-query-results( inverse-document-frequency( term ) )
\# so that the score is only affected if there is a multi-word query (so it is useless with AND queries?)
(19 - 5 TF-IDF weighting)
TFxIDF (term, document) = term-frequency( term, document ) \* Inverse-document-frequency( term ) =
(1 + log( count-of-term-in-document ) \* log-base-10( Number of documents / Number of documents where term occurs in (DF (t))
- this is the best measure so far (most commonly used)
- the value increases both with frequency of term and with the rarity of word in a document.
score of a query
the sum of TFxIDFscores for each of query terms.
(19 - 6 cosine scores)
Vector space := dimension: (t1,....tn) where n is number of terms in corpus/index;
each document is a point in this space (x1...xn) where xi - number of occurrences of word ti in given document.
Queries: also vectors (t1. tn) ti - search term in query; each search match also represented in the same way.(that's more practical)
distance between two vectors in in this vector space?
- euclidian distance: (not good if some search terms have high hit count); so vector has to be normalized:
- use angle between normalized vectors (vectors of the same length) (all xi divided by length of vector)
- so they use the cosine of the angle (cosine property: monotonically decreasing for 0 ... 180 degrees; only angles between 0 .. 90 - as all counts are positives.)
- for vectors x, y : cosine(x,y) = (X \* y) / ( |x| |y| ) \# linear alg: x \* y - dot product ; |x| - euclidian length of x.
- for search term (t1...tn) take vector q=(x1...xn) where x1=1...xn=1;
- for each search hit measure cosine between normalized vector for search hit and q vector; (smaller angle means higher value)
(19 - 7 use tf-idf and cosine scores together)
- there are many variants of term frequency weighting (taking logarithm of inverse term count is only one way of many);
- the SMART information retrieval system had a lot of influence here.
- how they compute the score (again):
- represent query as term frequency vector t1, ... tn - search terms of queries so this - its value is identity vector (each search term occurs once)
- compute inverse document frequency vector for each search term
- compute cosine between each search result and the identity vector (identity vector - which we have called the term frequency vector)
- sort search results by cosine value
(? position of a word in document is not evaluated: altavista used to assign higher ranking to documents with query terms in title; also no word of page-rank ?)
(19 - 8 evaluating search engines)
- want to measure user happiness / relevance of search results to users information needs (hard to quantify)
- so they assemble following test data:
- benchmark document collection and benchmark query collection
- by hand assign which documents are most relevant to a specific query
- run the system through this test set and measure precision/recall/F measure
- another measure: mean average precision (wants to consider quality of the first k top results)
- for each query in the benchmark set
- for each set of the first n query results (up to the limit k)
- compute precision of this set (\# relevant results / size of set)
- compute average over the computed precisions (this is called mean average precision)
- then average over all mean average precision results for all benchmark search queries.
(20 - 1 Word senses and word relations)
Some definitions:
lemma/citation form := same stem, part of speech
wordform := an inflected form as it appears in text
word sense := description of one interpretation/apsect of the meaning of a word
(word<index> : description)
Homonym := a word that holds multiple meaning;
Homophones := two words sound similar but have different meaning (Piece and peace)
Homograph := same word but holds different meaning (bat - club, bat - animal)
Polysemous word := word has multiple related meanings (bank - building of a bank; bank - financial institution)
Systemic polysemy := same pattern of different meanings (School,university,bank - can mean both building and institution) (this is also called systemic relationship
Test that word has more then one sense/meaning - zeugma test: construct a sentence that uses two meanings of the same word (construct the sentence out of two sentences with the same word)
"Does Lufthansa server breakfast and San Jose" - sounds weird, so there are two senses.
programs that process text have difficulty with ambiguity introduced by Homonyms.
two different words can share the same meaning (same meaning when the words are used in some specific contexts.
two lexemes have the same propositional meaning (are synonyms) if one can be substituted for the other in all situations (this is quite rare, so these are perfect synonyms vs. imperfect
synonyms that do not match in all situations (not acceptable in all situations)
(big,large synonymous when speaking about things (sense: older, grown up); large-sister - different meaning from big-sister (does not have meaning of older, grown-up)
Antonyms - two words with the opposite meaning (with respect to some particular meaning)
- can be at opposite ends of a scale (fast,slow)
- define a binary opposition (black white)
- can be reversed (or different directions) rise/fall up/down
hyponym := sense-a hyponym of sense-b -> if sense-a is in subclass of sense-b (car hyponym of vehicle)
(also denotes IS-A relationship between sets: A IS-A B (if x is in (is a) A it follows that x is in (is a) B) (another term: B subsumes A)
hypernym/superordinate := the reverse (vehicle is hypernym of car)
(20 - 1 Thesauri)
Thesauri/ontologies - repository for meaning (a categorisation that has the words as leaf nodes ? the categories are supposed to denote the meaning)
Wordnet:= thesaurus with aspect of dictionary. For each word there is a definition and its ontological relationship (part of speech; and IS-A/is instance of)
in wordnet a sense of a word is defined as a synonym set (set of words with the same description(gloss); these words would act as synonyms when used in a specific context, this context is defined y
the shared description/gloss)
Thesaurus := hierarchically organized lexical database (groups togather words by meaning)
Other relationships in wordnet :=
Hypernym, Hyponym, Member Meronym (Has-Member); Has-Instance; Instance-of; Member Holonym (Member-Of); Part Meronym (Has-Part) Part Holonym (Part-of); Antonym
Mesh := medical subject headings thesaurus
(he does not mention Roget's thesaurus)
(20 - 3 use a thesaurus to compute if words are similar)
word similarity := near synonyms
related words := linked by some chain of relationships as taken from thesaurus.
algorithms for computing similarity
- thesaurus based: look at the chain of links (length of the chain) between two words; do words have similar glosses/descriptions?
- distributional algorithms: look at statistical distribution of two words, are they similar?
thesaurus based/Passed based similarity - look at the length of the chain of relations that is linking two words
pathlen(s11, s2) = 1 + number of edges in shortest path in hypernym graph; where s1,s2 are two word senses/glosses.
simpath(s1, s2) = 1 / pathlen(s1, s2) \# want to make a value between 0 .. 1 (there is always some length != 0 because all senses are linked via the root category)
wordsim(w,w2) = maximum( foreach-word-sense-s1-ofw1( for-each-wordsense-s2-of-w2( simpath( s1, s2 ) ) )
Problem: each links are assigned the same distance; however links closer to the origin concept should have higher weight as they are more relevant, less abstract.
words( concept ) := look at subtree of words and concepts in thesaursus; take number of words and concepts in this subtree
Probability that a word belongs to a concept.
P( concept ) := words( concept ) / Number-of-words-in-corpus (? shouldn't that be sum of words and concept of a thesaurus ?)
Information-content( concept ) = - log ( P( concept ) ) \# just like in information theory
Lowest-common-subsumer( concept1, concept2 ) := lowest parent in thesaursus that is above both concept1 and concept2
Resik mmeasure: if two concept are linked via closer ancestor then they are more common.
Similarity-Resnik( concep1, concep2 ) = - log( P ( Lowest-common-subsumer( concept1, concept2 ) ) )
alternative Lin similarity measure (didn't quite understand that)
alternative: Lesk similarity measure: does not look at hierarchy, only looks at the description/glosses of concep1 and concept2
- for each sequence of words that appears in both gloss of concept1 and concept2 - add length(sequence of words ) ^ 2 to measure
- in some variations also compute this measure for hypernyms and hyponyms of concept1 and concept2 and add them to the resulting measure
Evaluation: intrinsic evaluation (compare with test set rated by humans); or extrinsic evaluation (plug it into a task and see if task performs better with new measure)
(20 - 4 word similarity - distributional methods to compute word similarity)
Thesaurus is not always available, or it lacks a given word/meaning, etc.
distributional methods do magic over vector spaces (like in information retrieval), however precision is lower then with thesaurus based methods.
Rational: if a word appears in the same context (near similar words) then it is counted as a synonym (Firth:1957 - 'You shall know a word by the company it keeps')
Intuition: word document matrix: rows of matrix - set of input words : we try to compute the similarity between these input words; columns: all documents; cells: count of word (column) in
document (row)
Now instehad of documents for rows we take more limited context: one passage or 10 words within a given word (? which set of words are chosen to form these contexts ?)
Also instead of word counts they are using the 'Pointwise mutual information' (PMI) measure - this measure is high when two event occur both, and low if not (events not correlated)
PMI( word-x, context-y) = log-base-2 ( P (word-x,context-y) / (P (word-x) \* P (context-y)) )
\# if word-x and context-y are correlated(word-x appears a lot in context-y) then its probability is higher and p(word-x, context-y) is higher
\# positive PMI (PPMI) - if logarithm is lower than 0 then return 0.
P (word-x,context-y) = (word-count\[ word-x\] \[context-j\] / sum-of-all-wordcounts-in-matrix) / P (word-x) P (context-y)
P (word-x) = sum-of-word-counts-in-row-for-word-x / sum-of-all-wordcounts-in-matrix
P (context-y) = sum-of-word-counts-in-column-for-context-y / sum-of-all-wordcounts-in-matrix
\# PMI has bias towards infrequent events, so use +1 smoothing (or +n smoothing)h
- like in search engines: use arc cosine to measure distance between these normalized vectors.
(20 - 5 word similarity - distributional methods 2)
better way, two ways are similar if they appear in similar parse context: (similar prepositions; similar head words, similar dependents)
so columns of matrix look as follows: subj-of~~<span style="text-align:left;">word> modifier-of</span>~~<word> ; obj-of-<word> etc.
- like in search engines: use arc cosine to measure distance between these normalized vectors.
(21 - 1 question answering)
question answering (1961:Simmons) ; parse question into dependency graph/tree; find matches of this dependency tree in database;
IBM Watson (2011); Siri (can ask factoid questions: Where is the Louvre? bit more complex one "do i need an umbrella tomorrow?); Wolfram Alpha")
two approaches:
- Information retrieval approach :
process question:
- try to detect answer type; domain of question (which one of the available data sources can answer the question)
- build information retrieval query
passage retrieval (information retrieval step)
- fetch query result and break up into passages.
answer processing
- extract answers from search result (ppassages) and rank answers
- hybrid methods
build shallow parse tree (constituent noun/verb phrases but do not parse the whole thing); use that for the usual query type detection;
use question to formulate multiple queries to information retrieval system (find candidates); so they have a lot of candidates
evaluate the answers by correlating it with specialized databases (geospacial DB; thesaurus/taxonomies; use temporal reasoning)
(use knowledge based methods for scoring answers)
parse tree/dependency tree of the question; build semantic representation for it; and find answer based on that methods to score answer
the system has semantic representation of questions that it understands/is able to answer; and it matches the parse tree of the input question to one of these.
(21 - 2 Information retrieval approach: step of processing input query: answer type detection)
- answer type detection
what kind of answer is expected in the question; does it want to find out a person name or place name? etc. etc.
A taxonomy of questions is used here (Li,Roth 2002)
6 coarse classes (ABBREVIATION,ENTITY,DESCRIPTION,HUMAN,LOCATION) these are divided into 50 finer classes (example: ENTITY: animal, body,color,currency (? what kind of taxonomy is
this?))
For IBM Watson they found 2500 answer types; most frequent 200 types covered <50% of Jeopardy questions.
can be done by hand written rules with regular expressions (who {is|was|were|are} PERSON)
first word of a query is often head word of a noun phrase (which city hots the olympics? -> which ; look at dependents of head word.
machine learning: train classifiers from training data (to assign one of the question categories from a taxonomy)
- query formulation
extract query that is to be passed to information retrieval system (aka. search engine)
as usual: heuristics x
- remove all stop words (a, the)
- in quotes - good indicator that this is part of query
- NNP words in recognized named entities
- nominals with adjective modifiers
- nouns
- etc. etc.
Any selection of these combinations (from first - highest priority to last)
- question type classification
is geo spacial data requested, or is it about restaurant reviews?
(if yes use specialized database, or specialized search engine for information retrieval; etc)
- focus words
which words within the question will be replaced by fact that is returned as the answer? (for formulating an answer)
- relation extraction - between parts of the question; these relations extracted are logical constraints that are used to filter out/evaluate an answer from search results
(21 - 3 passage retrieval + answer evaluation)
processing of query results returned by search engine:
- partition into shorter sections (paragraphs)
- check if any of the passages contain an answer to the question
Features tocheck:
- \# of named entities of requested type in passage
- \# of query words
- \# of n-grams from question
- proximity of keywords
- longest sequence of words from question
- etc. etc
evaluate is a passage contains an answer to the question
- can use named entity tagger: extract words with same tag as requested answer
- problem: a passage contains multiple answers; (again train classifier etc. etc.)
In Watson they have 50 different components; each one responsible for heuristics regarding one type of questions
for remaining ambiguities they use a magic ranking system to sort it out (mean reiprocal rank - did not quite understand that one)
(21 - 4 using knowledge in question answering)
- relations that have been extracted from wikipedia born-in(x,y) author-of(x,y) ; etc.
- temporal reasoning: when did AAA live? as a hint if AAA is a good answer to a question
- geospacial db: (which country is to the north of Vietnam?)
There are systems (SIRI) that allow reference to conversational state (him, who? - was mentioned earlier in the conversation)
(21 - 5 complex question answering)
(? during the end of an NLP course they usually start to do a lot of hand-waving, isn't it ?)
(22 - 1 summarization)
summarization: given a text A; produce a text B that contains information from text A that is important to the user (B is an abridged version of A)
this can be used for the following purposes
- create outlines/abstracts of articles (reddit has this!)
- summaries of email conversations
- extract action items from meetings
- simplifying of complicated text (?is this real?)
Several kinds of summarization
- summarization of single documents
- summarization of a set of multiple related documents
other subtypes of summarization
- generic summarization
- query focused summarization: extract information relevant to a query
used in google to display a short text about the search result
(often this is extracted from a single document);
- however in order to answer complex questions we might need to combine information from multiple document (and that is multiple document summarization)
two options to do summarization:
- extractive: pull out some sentences from the original document to produce the summary (google search results)
- abstractive: produce a summary with different wording from that of the source document (very difficult, we don't handle that here)
Abstractive summarization:
- one option is to take the first sentence from a document (good baseline algorithm) (and to mark terms of query as bold)
(22 - 2 generating snippet summaries)
stages:
- content selection:
- segment text into sentences (find sentence boundaries);
- pick those sentences that are relevant
- information ordering:
- select some most relevant sentences from those picked during previous stage
- choose the order of the resulting selection
- sentence realization
- might change the result somehow (might even simplify them sentences!)
Most search engines do only content selection:
First algorithm from Luhn:1958 - choose the documents that have informative words
- can use TfxIdf (search-term, document) score (see 19 - 5 lecture)
- or use log-likelyhood ratio measure (and he does not explain what that means ! explained in Dunning(1993) )
so choose words that log-likelyhood ratio is > 10.
- adapting this algorithm for query summarization:
weight-of-sentene := ( for all words in the sentence compute the sum of the following values:
if word in query -> return 1
if log likelyhood ratio of word > 10 -> return 1
else -> return 0
) / Number-of-words-in-sentence
Alternative: supervised content selection; train one over training set
features that can be used to train such a classifier:
- position in sentence
- length of sentence
- word informativeness
Problem: hard to get good training data; Humans when they write abstract often reforumlate and choose different word sets.
(22 - 3 Evaluating summaries)
- hard to evaluate answers of sumarization (mentioned before); x
- ROGUE metric: checks what percentage of bigrams from machine generated summary appear in test benchmark summary written by a human (bla bla bla)
- this is cheaper than having a human rate the answers generated by the machine.
(22 - 4 summarizing multiple documents)
- this is an active research topic; there is a competition for that (DUC 2005 competition)
- example question: "What has caused train wrecks in recent years and what has been done to prevent them?"
two approaches:
-bottom up snippet method
treat complex question as a single question;
Search and find relevant documents; get informative sentences and order and modify these sentences
More details:
Sentence segmentation of multiple documents
Simplify these sentences
- parse sentences;
- simplify parse tree by deleting some subordinate phrases;
- what can be deleted? delete some of the modifier clauses/subordinate sentences/preposition phrases that do not refer to named entities/ adverbials (for example,etc)
- rules decide what to delete from the parse tree.
- reassemble sentence from simplified parse tree
extract relevant sentences (again by computing weights like log likelyhood, etc; and picking those with highest weight)
- for all sentences: choose sentence with maximum relevance and add it to summary
- it has to have high cosine similarity with the query
- it has to be novel (so that chosen summary has no repetitions
so it has to have lo cosine similarity with text that has been selected to act as summary at the current moment
similarity-mmr( sentence, query, current-summary ) = max( lambda \* similarity( sentence, query ) + (1 - lambda) similarity( sentence, current-summary )
(mmr stands for maximum marginal relevance)
information ordering
- harder to do with multiple documents;
- can use chronological modification date of document returned by search and order by that
- order by topic: if documents by search are classified by some keywords
Information ordering - Information extraction method
- have a taxonomy of questions; for each question type define features that make a good answer to that question
(biography answer has: birth/death, education, why is he notable; nationality, etc)
- for each question type build a classifier that can extract sentences that contain a desired aspects; then combine these sentences into an answer.
sentence realization (?not explained what that is?)
-top down information extraction
- break up the complex question into component questions (component questions may be of different types)
- query for each component question
- order so that similar sentences have similar cosine measure (); or order so that adjacent sentences have same entity mentioned (talk about the same things)
</word></word></index></preposition></verb></verb></last></classname></count-of-docs></count-of-docs></count-of-docs></count-of-docs></code></blockquote>
Steve Yegge on software politics
2014-11-09T00:00:00+00:00
http://mosermichael.github.com//all/blog/2014/11/09/steve-yegge-politics
["", "blog"] | <notextile>Steve Yegge on software politics</notextile>
<hr />
<p class="publish_date">
9 November 2014
</p>
<p>Steve Yegge <a href="https://plus.google.com/110981030061712822816/posts/KaSKeg4vQtz">says here</a> that there are two types of approaches to software development.</p>
<p>The ‘conservative’ (risk averse) approach: bugs are not allowed to happen, quality comes first; Statically type checked languages are preferred</p>
<p>The ‘liberal’ approach (flexible design and speed to change come first): bugs are a fact of live so lets face them as they happen; Dynamic languages are tolerated. Are real world politics connected to this? Not significantly; the author says that more important are colleges and teachers + early (formative ?) experience as a programmer.</p>
<p>Software ‘conservatives’ say:</p>
<ul>
<li>bugs must be fixed before launch (like software specs, UML modeling and unit tests)</li>
<li>limit syntax to ‘safe’ features (‘central committee’ approves style guide that forbids code generation/internal DSL, dynamic RTTI, exceptions, etc).</li>
<li>static type checking a must</li>
<li>data must be stored according to well defined schema (in third normal form)</li>
<li>no debugging features on production system (to avoid back-doors)</li>
<li>fast is better then slow</li>
</ul>
<p>software ‘liberals’ say</p>
<ul>
<li>bugs are not a big deal; fix them as they happen</li>
<li>fewer rules, so that there is a short learning period for new programmers (not one year like in Google)</li>
<li>don’t need protection from language features;</li>
<li>code should be small (shortness conveys expressiveness / ‘power’ of an abstraction )</li>
<li>requirement of data schema slows down development. Schema should follow the code - when nature of data is better understood.</li>
<li>interfaces: Modeling is just guessing how interface will look like. Possibility to evolve an interface is valued (also backward and forward compatibility</li>
<li>Public interfaces should always do the simplest thing that could possibly work, and grow only as needed. ‘</li>
<li>System flexibility can mean the difference between you getting the customer (or contract) vs. your competitor nabbing it instead. Security and safety risks in run time production systems can be mitigated and controlled by logging, monitoring and auditing. There are plenty of existence-proofs of large systems with root-access backdoor and shells (e.g. RDBMS, online game servers) whose risk is controlled while still giving them world-class run time flexibility.</li>
<li>Premature optimization is the root of all evil. Get the code working first.</li>
</ul>
<p>Examples of ‘software conservative’ companies: Google and Microsoft (very conservative)
Examples of ‘liberal’ companies: Facebook and Amazon</p>
<hr />
<p>I think this analysis is very interesting: it tries to characterize organizations in a way that is independent of the size of an organization: there can be software conservative start-ups and software liberal big mega corporations;</p>
<p>Now i think that its business that is pushing a companies into the camp of ‘software liberals’ or ‘software conservatives’. A company where the user experience is important will need to evolve the UI quickly; sometimes the feature set/time to market is more important than software quality. Such a setting will lead to a ‘softaware liberal’ approach.</p>
<p>For example Facebook had to climb to the top of the very dynamic ‘social’ heap, this extensive growth model made them very software progressive. At an early stage it was probably very important to evolve the product speedily; the winner would need a more dynamic setting where good ideas would be adopted and bad ones discarded; Here quick turnaround of features was very important.</p>
<p>(well they need to stay on top of the ‘social heap’ in order to stay relevant, a big established platform has often other requirements, but apparently the founders are still dominant here)</p>
<p>Google and Microsoft both do stuff that must work reliably in complex settings, so they are not likely to experiment with dynamic languages (at least not for the ‘core’ product). Also both companies must preserve their relative positions, so they are more defensive and have more to loose/fear to loose their relative positions, this makes them more ‘software conservative’. (both Google and Microsoft are perceived to have this one great product that everything else depends on)</p>
<p>Also Microsoft (or IBM) products have a strictly defined and managed product cycle; this makes for less room for experimentation; bugs have to be avoided and can’t be accepted as a fact of live; Google had to be good enough to do without much customer support, so this requires a high level of quality up-front (all for different reasons, see below); strong quality requirements make them much less tolerant of bugs.</p>
<p>An interesting question is if a ‘software conservative’ company has a more complicated management structure than a ‘software liberal’ company.</p>
<hr />
<p>It is often fairly limiting to evaluate real world politics according to a one-dimensional scale of left versus right. One way to look at things is to add additional dimensions: a second scale in addition to left<-<del><span style="text-align:right;">right is authoritarian<</span></del>>libertarian.</p>
<p>Let’s see if this second dimension can be added to the analysis of software companies, I think it is independent of the political dimensions, there can be strictly top-down managed software liberal and software conservative companies.</p>
<ul>
<li>authoritarian shops would be strictly hierarchic organization where business decisions are made by the top level, design documents and schedules formalized by middle level management and then passed down to be executed by grunt programmers. Even this approach has gradations: during design stage: is feedback encouraged from lower levels, or is it the norm to brush such feedback away? Is criticism tolerated? On the one hand these factors are determined by the personalities of managers within an organization, on the other hand a manager with a style that does not fit into the prevailing management culture is not likely to stay there for a very long time. How is information shared within the company? Are decissions just passed down the hierarchy or does management care to share contextual information about decissions and feature schedules (to be updated on a need to know basis is always less than what is required for a full understanding) ? The amount of information that is shared within an organization is an important factor. informed and constructive feedback from below would have to be based on some real factual knowledge. Important questions here are: does management try to act as an effective shield of information, are lower ranks informed on a need to know basis? What is the amount of interaction between peers? Is there a culture that encourages information sharing between peers or is it of benefit to hide knowledge in order to maximize job security? Hierarchy is often enforced by subtle gradations: so authoritarian shops are more likely to stress distinctions and boundaries between ranks. It is interesting that large authoritarian organizations do have research divisions that are organized with a reduced measure of hierarchic control.</li>
<li>Goggle (pre Google<em>) is said to have been a very non authoritarian work place. Initiative from below was encouraged, often big initiatives started as personal projects (the 10% project). Unfortunately this could only have worked when they had this playground where they could try out new things: Social networks like facebook were recognized as threats to the business model and Google decided to consolidate around the Google</em> platform. A platform means that every feature has to fit in with the rest of the system, so there is less room to try out new thing, this change also comes with more hierarchic style of product management; is also slashes the digital playground - a platform does not tolerate contradictions/overlap between features. I would guess that information sharing also runs into problems as a company grows into a mega corporation. It is interesting that Google’s lack of customer support is what enabled this style in the first place: (google products are a take-it or leave it proposition). For developers there is much less time for trying out new things when one is frequently interrupted by customer problems.</li>
</ul>
<hr />
<p>On the personal side there is one major trade off:</p>
<ul>
<li>Working for bigCorp has its benefits - the main benefit is stability, the downside is that one has to put up with a log of hierarchic crap.</li>
<li>Start-ups have less hierarchy but you have to put up with less stability (political infighting between founders, question of funding, question of direction)</li>
</ul>
<p>Nothing is perfect; coding is only part of the problem when working in software ;-)</p>
<p>i am not the only one to suspect that half of the complexity of any given software product is due to internal politics <a href="https://en.wikipedia.org/wiki/Conway%27s_law">here</a> - in the end its always the guy who is handling grep and ctags and who has to figure this shit out …</p>
<center>
<img src="/cstuff/positive.jpg" />
</center>
By the rivers of Babylon
2014-11-07T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2014/11/07/rivers-of-baylon
["", "ramblings"] | <h2 id="by-the-rivers-of-babylon">By the rivers of Babylon</h2>
<p class="publish_date">
18 July 2011
</p>
<p>i have read some publicly available sources on the history of Sumerian and Babylonian civilization; (most articles are behind paywalls) (my notes on the sources are at the end of this text)</p>
<p>some things I have learned (i am not a specialists so my deductions may be all wrong ;-)</p>
<h3 id="common-themes">common themes</h3>
<p>Mesopotamia developed an irrigation based intensive agriculture (10:1 ratio of yield to seed); the King/palace would construct the irrigation system (use of corvee labor) while the temple would be responsible for its maintenance; These centers managed surpluses so as to enable production of merchandise, this merchandise would then be traded for valuable raw materials (the south was lacking metals; timber and stones).</p>
<p>Long distance trade was the motor of the bronze age economy; also this was often a risky business that was left in private hands (commercial credit was at 20% interest. so the risk must have been high; if robbed a merchant’s debt could not be recalled; other risks: weights would be manipulated and it is easy to dilute silver); However merchants were independent of the crown (even in UR III times) and could make their own fortune.</p>
<p>In most cases there were several co-existing centers of power, so the palace power was limited by the power of the temple and the local gentry + merchants (an on some occasions city assemblies).
Also In early times city assemblies might have had an important role: see both the ‘Gilgamesh Epos’ and Babylonian ‘Epic of creation’ (<a href="http://www.fofweb.com/History/HistRefMain.asp?iPin=MESP0167&SID=2&DatabaseName=Ancient+and+Medieval+History+Online&InputText=%22kingship+in+ancient+Mesopotamia%22&SearchStyle=&dTitle=kingship+in+ancient+Mesopotamia&TabRecordType=All+Records&BioCountPass=0&SubCountPass=1&DocCountPass=0&ImgCountPass=0&MapCountPass=0&FedCountPass=&MedCountPass=0&NewsCountPass=0&RecPosition=1&AmericanData=&WomenData=&AFHCData=&IndianData=&WorldData=&AncientData=Set&GovernmentData">here</a> ).</p>
<p>On a few occasions kings proclaimed themselves as gods (Naram-Sin of Akkadia and Shulgi in UrIII); I guess that in these instances that the palace was trying to take some power away from the the temple (at least in those city-states that turned into regional powers). (sacred marriages between kings and goddess don’t count here - these are supposed to be fertility rites)</p>
<h3 id="not-quite-despotism">not quite despotism</h3>
<p>For most of the time the region does not seem to have fit into the model of a <a href="http://en.wikipedia.org/wiki/Hydraulic_empire">Hydraulic civilization</a> with an <a href="http://en.wikipedia.org/wiki/Asiatic_mode_of_production">Asiatic mode of production</a>
This model was proposed by Marx and Wittvogel: they state that the requirement for a complex system of irrigation can only be met by concerted effort of a great number of people, this created a manager class who would gain monopoly status and grab unrestrained absolute power, leading to permanent despotism.</p>
<p>For most of the time Babylonian society seems to have been a legalistic society (like the middle ages in Europe) so that property rights were likely respected (hard to say: Urukagina reform documents decried abuses such as looting of the common man by the elites).</p>
<p>At any given time they had free households; temple/palace dependents and slaves, i guess it is hard to tell which form was the dominant ‘mode of production’ at any given time.</p>
<p>(see my notes on “Oriental Despotism and the Asiatic Mode or Production: A Modern day critique” Alexandria Alfrano <a href=”#mrx”>here</a>).</p>
<p>Some of the arrangements in such a mixed economy were surprisingly modern
(my notes on “Neo Babylonian Entrepreneurs’ by Cornelia Wunsch <a href=”#cwns”>here</a>)
(my notes on “‘Entrepreneurs: From the Near Eastern takeoff to the Roman collapse’ by Michael Hudson” <a href=”#hud1”>here</a>)</p>
<h3 id="state-collapse">State collapse</h3>
<p>Collapse of a state was brought about by a combination of factors: crop failures and invasion of nomads or defeat in a war; climate change/ecological change such as significant change of course of the Euphrates or increase in soil salination.</p>
<p>In times of hardship independent households would become dependents or slaves of the temple/palace so that these institutions also had a backup function of providing food security.</p>
<p>It might have been that Bronze Age society was very much aware of these immanent dangers; this may have brought about a sense of corporatism and co-dependence, a strong factor for cohesion of society .</p>
<p>During the first millennium BC the once dominant south became weaker - they say that was due to increased salination of the soil;</p>
<p>The temple and palace economy seems to have grown less dominant over time, (especially since the end of the Old Babylonian Empire and the dark ages that followed)
Since the end of the Bronze age private owners and the gentry seem to have gained in relative importance.</p>
<p>(My notes on “Collapse in Early Mesopotamian States: what Happened and What Didn’t” <a href=”#bum”>here</a>)</p>
<h3 id="debt-cancellation-and-renewal">debt cancellation and renewal</h3>
<p>Another peculiar feature is the tradition of debt cancellation; between 2400BC and 1400BC some thirty debt cancellations.
(also in Jewish law there has been the feature of debt cancellation; this was also practiced in Egypt and rarely in Greece (by Sauron))</p>
<p>Michael Hudson sites the following reasons for this policy:
* preservation of society; if all subjects end up as tax slaves then there would have been no army conscripts (need freeman for that)
* to assert dominance of the palace/temple economy over emerging private owners
Debt cancellation was practical when most of the debt was owned by the palace or temple; it was no longer an option once private owners began to dominate.
* were explained as acts of renewal of nature (like the Shmita laws)</p>
<p>Institutions were aware of systemic dangers and took corrective actions; i guess that a general feeling of bronze age instability must have helped to develop these practices.</p>
<p>(my notes on ‘The lost tradition of Biblical debt cancellations’ by Michael Hudson <a href=”#dbtc”>here</a>)</p>
<p>(Interesting that this policy resembles Roosevelt’s <a href="http://en.wikipedia.org/wiki/New_Deal">New Deal</a> ; also Michael Hudson who studied debt cancellations is no fan of the neo-con economists either)</p>
<h3 id="empires">Empires</h3>
<p>periods of centralized rule were the exception rather than the norm; Centralized states introduced important innovations, like the unified system of measures and weights under the Akkadian empire (weights for silver were the precursor of money) + better roads and messenger service and more secure communications.
(here they were similar to the <a href="http://en.wikipedia.org/wiki/Qin_dynasty">Qin dynasty</a> in China)</p>
<p>However trade was possible without centralized rule, as trade was in the mutual interest of all cities - due to the common lack of resources (there seem to have been norms to regulate trading).
Centralized empires would impose tributes which were very much resented by dependent city-states, so that scheduled rebellions would break out at times of succession (no equivalent for the ‘Pax Romana’ in these quarters).</p>
<h3 id="social-mobility">Social mobility</h3>
<p>A significant portion of the rulers who made a big impact where of humble birth or non royal lineage; <a href="http://en.wikipedia.org/wiki/Sargon_of_Akkad">Sargon</a>
‘Urukagina’ <a href="http://en.wikipedia.org/wiki/Ur-Nammu">Ur-Nammu/Ur-Engur</a> <a href="http://en.wikipedia.org/wiki/Tiglath-Pileser_III">Tiglath-Pileser III</a> ; it seems that there has been a significant degree of social mobility (probably this is another sign of general instability/uncertainty)</p>
<p>In early Mesopotamia the people could replace a ruler who was perceived to lack ‘heavenly sanction’ (that would mean famine or defeat);
(similar: In Egypt the rulers derived their authority as the upholders of - <a href="http://en.wikipedia.org/wiki/Maat">Maat</a> - the principle of law, justice and harmony where the goddess of Maat would have a continuous role of preventing a slippage of the world into chaos) In Mesopotamia heavenly sanction was to be endorsed by the Enlil temple at Nippur - so the priests there were of regional importance .</p>
<p>I would say that these are all signs of a deep rooted perception of insecurity that seems to have been an important feature of the bronze age.</p>
<h3 id="temple-as-the-backup-of-civilization">Temple as the backup of civilization</h3>
<p>The Temple seems to have been the backup plan for civilization, on the event of system breakdown the temple seems to have taken care of the continuity issue.
For example the Old Babylonian empire was conquered by Kassites (and later Mitanni) the new ruler adopted part of the old system (like the Marduck cult) .</p>
<p>Compare this to states where the ruler claimed divine title (Akkadian empire, Ur III; also in Neo Assyrian times the crown became much stronger at the expense of temple (however Ashurbanipal is supposed to have been a pious ruler)) - when these states where defeated their culture was gone for good.</p>
<h3 id="lagash">Lagash</h3>
<p>The reforms of Urukagina of Lagash: (my notes on L.W King ‘History of Sumer and Akkad’ <a href=”#lwk”>here</a>)</p>
<ul>
<li>
<p>the big change came in the second half of the third millennium BC (the early dynasty III period); the writing system settled into syllabic cuneiform signs; historic events were now recorded.</p>
</li>
<li>
<p>The early rulers of Lagash were rather modest; they regarded themselves as the shepherds of the city deity and had a small household</p>
</li>
<li>
<p>Lagash turned into a regional power and tributes would prop up the business; as a result the temple and palace administrations became bloated as they filled up with ex-military personal.</p>
</li>
<li>
<p>harder times followed (marked by a quick succession of rulers), so these officials turned towards looting of their own population.</p>
</li>
<li>discontent grew and Urukagina gained power (an outsider of non royal linage, might have been voted in by the city assembly)</li>
<li>Urukagina started a staged reform program:</li>
</ul>
<p>First stage of reforms: correction of ‘isolated shortcomings’; a tablet enumerates problems that existed “since the beginning”.</p>
<p>* Corruption from the top down, also the temple and crown - one hand was washing the other.
* Natural that with extended power of Lagash came an elaborate court, lots of temples and an army of officials:
“But with the growth of her power as a state, she lost many of the qualities by virtue of which her earlier successes were achieved. The simplicity, which characterized the patesi’s household at a time when he was little more than a chief among his fellows, was gradually exchanged for the elaborate organization of a powerful court.”
in less fortunate times the taxes to maintain all this were quite a strain…
“Urukagina records that his predecessors on the throne had appropriated the property of the temples for their own use. … The priests themselves grew rich at the expense of the temples, and plundered the people with impunity.”
* Palace began to usurp/overshadow role of the temple - appropriate temple property for their use; Palace priests would steal from commoners and divide spoils with palace.
Measures:
* return palace land to temple.
* installed cult of goddess Bau
* proclaims that his own aim is to be administrator in the interest of the city god.</p>
<p>Second stage of reforms: “attack the abuses which existed among the secular officials and the priests”
* Sacks officials/inspectors, sacks officials who accepted bribes from
* priests; high burial fees were cut down by half (? fees include a kid a bed and a seat ??); taxes/fees were cut
* in ancient regime extortion and robbery were practiced by the powerful - they could take anything at will ! ‘sought to protect by law the humbler classes of his subjects from oppression by their wealthier and more powerful neighbors’
The powerful now had to buy things for a price, and the owner could refuse and may not be molested (rights of property !)
* Fees for divination were abolished (Shekel was the currency !) (Former; one shekel for diviner, one for priest, one for Patesi - cut down to one shekel)
* abolished divorce fees (this measure protected women, as payment of five shekel (a shekel was a monthly salary in old babylonian times) was required to get rid of the wife (that would be quite a lot of money))</p>
<p>Laws of Urukagina are similar to laws of Hammurabi. Both drew legitimacy of laws on being given by the(pr local deity; both say said they were the champion of the poor against the strong.
(well slavery was maintained by both of them) “In his reign, he (Urukagina) says, to the widow and the orphan the strong man did no harm.”</p>
<p>However few references to other city-states (except for ties with Nippur and rebuilding of Enlil temple); still the city walls were rebuild.
clear that main interest were internal matters. Construction projects: temples, city walls;
(? <a href="http://en.wikipedia.org/wiki/Urukagina">Wikipedia</a> says that the temple of Bau increased its dependent from 50 to 1500 - so some heavy redistribution seems to have taken place ?)
“The host of officials he abolished or dispossessed of office had belonged to a military administration” … “When war threatened he must have found himself without an army and without the means of raising one.”</p>
<p>Some points of interest:</p>
<ul>
<li>
<p>only L.W. King describes the law codex as a measure designed to protect the poor against abuse</p>
</li>
<li>
<p>the King/palace seems to have tried to take away from the temple as the state became a regional power</p>
</li>
<li>the story of Pharao <a href="http://en.wikipedia.org/wiki/Akhenaten">Akhenaten</a> (about a thousand years later in Egypt) has similar themes with the Urukagina reforms; both cited disgust with corruption as triggers for reform, both seem to have neglected foreign relations in favor of internal reforms; both movements have alienated significant parts of the native establishment.</li>
<li>this description has some striking similarities with Perestroika, which also came in stages (1985-1987 correction of ‘isolated shortcomings’; 1987-1989 reforms with the aim of transition to ‘democratic socialism’; 1989-1991 destabilization)
There seems to be nothing new under the sun.</li>
</ul>
<h3 id="ur-iii">UR III</h3>
<p>The third dynasty of UR is often described as a very strict and despotic state with a huge bureaucracy (some 100.000 tablets of (mostly) administrative records have been found) where the economy was under exclusive state control. Nowadays this is not so sure - most of the patterns of Mesopotamian economy seem to have applied to UR III. Also prices of barley from private producers rose sharply when the empire was in the process of disintegrating, so there must have been independent producers.</p>
<ul>
<li>
<p>Ur III turned into a regional power; King Shulgi declared himself a god (must have been part of a power grab)</p>
</li>
<li>they installed their own administrators in conquered cities (similar to the Akkadian empire)</li>
<li>there was a stress on ideology: the Sumerian kings list was compiled in order to justify the rule of the current dynasty and this is the first time that the Gilgamesh Epos was recorded.</li>
</ul>
<p>Still the administration of Gudea would probably have been quite different from that of Shulgi.</p>
<h3 id="old-babylonian-empire">Old Babylonian empire</h3>
<p>The old Babylonian empire is often described as a high point of literacy and learning; Some say that it collapsed with a big crash: Samsu-iluna fought insurrections by diverting the Euphrates - this is said to have caused soil salinization in large areas so that southern Babylonia turned less important for centuries (? that does not explain how Babylon remained to be the religious center of the region?)
Some say ( <a href=”#bum”>here</a>) during the bronze age they still lacked the capacity to change the environment in such a way)</p>
<h3 id="neo-assyrian-times">Neo Assyrian times</h3>
<p>What kind of place was the Neo-Assyrian empire? The first millennium BC saw the introduction of the ‘strong’ centralized state - with a standing army and strong bureaucracy the state would control all aspects of live directly, without the need for relatively independent vassals.
it is sometimes seen to have been a very nasty place; (all ‘strong state’ seem to be like this); Heavily militarized with a standing army and heavy handed bureaucracy: land ownership was conditional on military service (ilkum service); women were now seen as inferior, locked up at home and had to wear veils (this feature stuck in the region) (<a href="http://www.fofweb.com/History/HistRefMain.asp?iPin=ESCAW675&SID=2&DatabaseName=Ancient+and+Medieval+History+Online&InputText=%22social+organization+in+ancient+Mesopotamia%22&SearchStyle=&dTitle=social+organization+in+ancient+Mesopotamia&TabRecordType=All+Records&BioCountPass=0&SubCountPass=1&DocCountPass=0&ImgCountPass=0&MapCountPass=0&FedCountPass=&MedCountPass=0&NewsCountPass=0&RecPosition=1&AmericanData=&WomenData=&AFHCData=&IndianData=&WorldData=&AncientData=Set&GovernmentData=">here</a> )</p>
<p>Heavy tributes/taxes would cause frequent uprisings, is the strong state really more stabile than loose alliances of states?
Assyria was not densely populated; the successful rulers would therefore resettle a huge number of people by force (the Akkadian empire is said to have done the same)
(in contrast to that: Neo-babylonian Nebuchadrezzar II deported the elites of Israel, not the whole population)</p>
<p>Others see it as a multi ethnic empire; a place of technological innovations and high culture/learning (the <a href="http://www.atour.com/education/pdf/SimoParpola-AssyrianLibraryRecords.pdf">library of Ashurbanipal</a> )
i guess a lot would depended on the personality of the ruler; the administration of Sancherib (who sacked Babylon) must have been quite different from that of Ashurbanipal.</p>
<h2 id="assyriology-and-modernity">Assyriology and modernity</h2>
<p>The 1860ies saw a new interest in the ancient history of Babylon; from ‘Ancient Mesopotamia / New Perspectives’ :</p>
<p>“The later nineteenth century was a time of great intellectual upheaval. The developing science of geology was revealing the immense age and gradual formation of the earth, while Charles Darwin was showing how life had evolved in all its diversity. This new knowledge undermined the certainties of the Bible, according to which the world was created in immutable form at a date calculated by biblical scholars as 4004 B.C.E. In this epoch of challenge to established traditional views, many found it reassuring that archaeological research in the Near East was uncovering cities and records of individuals familiar from the Bible, thus confirming and buttressing its authenticity.” (notes on ‘Ancient Mesopotamia / New Perspectives’ Jane R. McIntosh ‘<a href=”#mod”>here</a>)</p>
<p>The early phase of excavation was also marked by an intense international competition for new findings; I noticed that L. W. King always refers to the ‘German excavations’ he does not refer to the respective scholars by name.</p>
<p>Up to the early twentieth century Assyriology seems to have had relied on the study of written records; also excavation techniques were rather crude.
It turns out that an awful lot can be learned from studying auxiliary information derived from the style of pottery and layout of houses found at a given layer - the technique of Stratigraphy Robert Koldewey was the first one to adopt this knowledge in the region during his excavations of Babylon in 1899 <a href="http://cchag.org/html/09476/iraq03iraqenl.html">here</a></p>
<p>Now thanks to this thorough approach the Pergamon museum can show off with the Ishtar gate, other museums have interesting exhibits, however this is all not quite the same experience.</p>
<iframe width="420" height="315" src="//www.youtube.com/embed/OldisweN8QQ" frameborder="0" allowfullscreen=""></iframe>
<p>(Modern archeological techniques were first perfected by <a href="http://www.britannica.com/EBchecked/topic/454163/Sir-Flinders-Petrie">Sir Flinders Petrie</a> in Egypt and Palestine in 1890; a bit before Koldewey)</p>
<p>It is interesting to note how Archeology changed into its present state after the Second World War: it is an increasingly multidisciplinary field where each tiny piece of information is systematically analyzed,</p>
<ul>
<li>all with the intention of understanding the given society as a system of related parts, as a whole. These changes originated from a perfection of excavation techniques; This trend also coincided with a major change in historiography (the way that History is understood); here there was a marked shift away from political history towards social and cultural history.</li>
</ul>
<p>I may be wrong but there seems to be an additional factor that triggered these trends: the practice of military intelligence gathering during the war;
Here the opposing armies turned into very complex systems, so it goes that Military intelligence would turn into the study of complex system; it had to
* study the communications exchanged between enemy units of various sizes
* deduce the operating procedures and the immediate intentions of the adversary from its pattern of activity and intercepted communications; here Operations research was invented
quite a task given that an army consists of a hierarchy of units - lots of details here.
* the Strategic bombing survey had to study the enemies economic system in great detail, so as to identify the set of targets that would cause maximum damage.
Strategic intelligence would need to study the capabilities and intentions of the enemy from a global view.
* every scrap of information would be used: see the <a href="http://en.wikipedia.org/wiki/German_tank_problem">German tank problem</a> - here serial numbers of machine parts were used to create estimates of overall tank production.
Electronic warfar/Radars turned into a complex beast ( <a href="https://www.youtube.com/watch?v=ZTC_RxWN_xo&start=609">here</a> )</p>
<p>Many scholars from various fields of the humanities were also mobilized into the war effort, many were active in intelligence work, and it is possible that methods of systems research would show its influence on their field of study after the war ended.</p>
<p>It seems that our present day biases and preferences all have a strong influence on how we understand the past.</p>
<h2 id="conclusion">Conclusion</h2>
<p>We do seem to know very little for sure - Mesopotamian tablets were often cryptic (lots of signs are not understood - or context is not known); most what is known with some degree of certainty comes from Old Babylonian translations into Akkadian; the that whole field is subject to endless interpretations.
I found that is very easy to find points of contradictions between the different theories.</p>
<p>I think that the attempt of understanding the demise of ancient civilizations is similar to the ideas of how Dinosaur extinction: to some degree both are mirrors of our own fears.
The <a href="http://en.wikipedia.org/wiki/Alvarez_hypothesis">impact hypothesis</a> of dinosaur extinction became popular as we feared a <a href="http://en.wikipedia.org/wiki/Nuclear_winter">nuclear winter</a>
At the present there is the fear of <a href="http://en.wikipedia.org/wiki/Global_warming">global warming</a> so environmental factors are seen to have had major part in the demize of ancient civilizations (like the <a href="http://en.wikipedia.org/wiki/4.2_kiloyear_event">4.2 kiloyear event</a> and the role of the <a href="http://en.wikipedia.org/wiki/Hekla_3_eruption">Hekla 3 erruption</a> in the <a href="http://en.wikipedia.org/wiki/Late_Bronze_Age_collapse">bronze age collapse</a> )</p>
<p>Maybe all these data points will lead future observers to the conclusion that our age was one of deep rooted fears…</p>
<p>my notes on “Collapse in Early Mesopotamian States: what Happened and What Didn’t” ( “Source”:https://www.academia.edu/210387/Collapse_in_Early_Mesopotamian_States_What_Happened_and_What_Didnt )</p>
<ul>
<li>
<p>no common opinion on what is ‘state collapse’ (often there is a follow up order - like Rome transformed into the middle ages)</p>
</li>
<li>what caused collapse of URIII ?
<ul>
<li>Weiss: aridity caused migration of Amorites -> military challenge (climate confirmed by one source only)</li>
<li>Tainter: mismanaged of irrigation + abandonment of areas caused salination of soil + collapse of resource base</li>
<li>(author) other city states grabbed opportunity to shake of domination, they could do without the empire (and did not like to pay tributes at all)</li>
<li>still we don’t know much (tablets poorly understood)</li>
</ul>
</li>
<li>collapse of Uruk (abandonment in late OBE)
<ul>
<li>Uruk was an important city in Mesopotamia, provinical capital in Akadian empire,URIII and OBE.</li>
<li>most of the Uruk temple staff moved to Kish (!)</li>
<li>the author doubts that: abandonment is NOT due to salination that was brought about by diversion of Euphrates during campaigns of Samsu-iluna (says they did not have the technology, only the early Kaliphs could do that, two throusand years later)</li>
<li>author speculates : elites of Uruk aligned with Babylon; disaffected lower classes aligned with Sealanders and these depopulated the city once they have won (???)</li>
</ul>
</li>
<li>collapse of Assyrian empire
Other societies were heterarchical - both temple/palace/gentry+merchants+assemblies were alternative power center.
<ul>
<li>Assyrian empire was highly centralized (around palace); (palace/military/gentry were interconnected not independent) once the power center was gone nobody would pick things up;
(??? if they were so integrated then how comes that Sennacherib was killed because his sacking of Babylon was seen as impious ; also trading guilds seem to have played a major role ???)</li>
<li>wholesale deportations of populations would dilute the Assyrian base; so lower levels were no interest to re-create the power structure from bottom up (on system failure) ; no ‘re-generators’ left that could start regeneration of system (? in Persian times Assyria did revolt, but that was later ?)</li>
<li>Babylon was still seen as the religious/(cultural ?) center of Assyria.</li>
<li>still fall of Assyria did not lead to sharp reduction in population !</li>
<li>Kaufmann: tightly coupled systems are vulnerable because failure of one subsystem quickly cascades (loosely coupled system said to be more stable)
Tight coupling led to the success of Assyria - it was also what led to the downfall.
“scale-free/power law networks” : robust to random failure but vulnerable to attack on region with high connectivity <br />
.</li>
</ul>
</li>
</ul>
<p>my notes on ‘Entrepreneurs: From the Near Eastern takeoff to the Roman collapse’ by Michael Hudson (Source “here”:http://books.google.co.il/books?id=6rw1fxYalfUC&lpg=PA62&ots=43aGWB_zdC&dq=the%20invention%20of%20the%20enterprise&hl=iw&pg=PA9#v=onepage&q=the%20invention%20of%20the%20enterprise&f=false )</p>
<hr />
<p>first commerce: gift exchanges. subsistence/low surplus economies don’t like accumulation of personal wealth (this is seen to be at the expense of the community, and adversely affect chances of survival for the community)
Public norms were to spent surpluses: via gift exchange; public festivals or burial offerings. Status was gained by giving away wealth.
(However the chief is allowed to concentrate wealth; he is supposed to use it ‘on behalf of the community’ ) (? isn’t that bullshit ?)</p>
<h2 id="institutional-temple-economy-evolved-from-the-chiefs-household-in-third-millennium-bc-palace-came-a-bit-later-">Institutional temple economy evolved from the chiefs household in third millennium BC (palace came a bit later ?)</h2>
<p>Max Weber: drive for social status motivates economic motives; traditional economists and Marx say that the reverse is true - they did not see that public institutions provided capital and acted as catalysts for production.
The view was that money appeared spontaneously - to decrease transaction costs and as means of payment,value store and measure of cost.
historically economists did not look at Babylonia as the source of modern business practices - it lay outside the ‘western continuum’ - which was supposed to have started with classical Greece.
But now it is ‘ex oriente lux’ the light(culture) comes from the east.
—</p>
<p>The economic practice of antiquity came from Mesopotamia: uniform weights / money as measure of wealth; interest lending and profit sharing agreements (beer selling concessions); account keeping and reports; : the long distance trader was the first entrepreneur.
First used to accumulate surplus in order to trade for resources that were lacking in Mesopotamia (timber, stones, metals)
Temple/palace did not levy taxes; they had own endowments; but they leased out fields and workshops; advanced goods to traders for later payment (upon return from journey)</p>
<ul>
<li>the payment for lease was based on yield estimates; on success the surplus profit, on bad harvest the difference turned into debt.
Therefore it was a mixed economy (as merchants and tenants were semi-independent);
Damgar/tamkurum = merchant, entrepreneur; would accumulate their own capital
Prices evolved as a book-keeping tool - to measure debt obligations; prices were administratively fixed - this actually helped trade (provided for stable context for business)
Social status of merchants was high - well connected to temple/palace hierarchy.
Debt cancellations (or agrarian - barley denominated debt, not commercial ‘silver’ debt) - to protect society from debt based forfeiture and to preserve the army (infantry was from among landed peasants) (Babylonian
Even commercial law freed merchants from liability - on event of shipwreck or piracy (would not be compatible with Roman ideas of law)</li>
</ul>
<p>In Assyria state institutions played a lesser role; Trade ‘houses’ were in charge; guilds represented merchants interests with the authorities.????)</p>
<p>commercial loans to finance trade: combined interest bearing debt (20% - due to high risk) with profit sharing partnership agreement;</p>
<p>Hammurabi’s laws: merchant were to split profits 50-50 with their backers in the palace/temple
If a merchant pays advance to a trading expedition:
if expedition makes a profit -> has to pay back advance with interest to merchant
no profit -> pay back twice the advance
loss -> still has to pay back advance
shipwreck or piracy -> no liability !
Had promisory notes: (seem to have been transferable obligations; this would have been a complex instrument)
OBE: Agricultural debt for rents of land or payment for concessions (for beer trade of ‘ale women’) was to temple/palace;
Hammurabi’s law: if land was flooded then no obligation to pay rent !</p>
<p>(???? in Neo Babylonian times: in the next article they say that the Egibi famility did tax farming - exactly like the Roman outsourcing model ! maybe in succeeding times private owners gained in relative importance ????)</p>
<hr />
<p>These commercial techniques were exported to Mycean Greece in second millennium BC; later reintroduced to the west around 750BC (standardized weights, financing/interest+debt, money) - the west had less institutional checks & balances (lack of debt cancellations - except for Solon).
the Roman/Greek elites were detached/rent seeking + viewed commerce with disdain (delegated affairs to subordinates); also lack of debt cancellation led to polarization + debt dependency.
Roman law held that property rights were absolute + were forfeited irreversibly; debt became mechanisms for centralization of land ownership (transfer from self supporting households to absentee owners); the law was creditor oriented.
hereditary landed wealth + consolidation of big estates (due to debt forfeiture of small tenants) leads to corrosive forms of enterprise: the landed gentry gets detached from/above commercial activity.
Money lending was looked at with disdain, but became a major tool that led to consolidation of estates.</p>
<p>In Rome the source of wealth were predatory practices, these were seen as noble. ‘the oligarchies ethic preferred seizing wealth abroad to creating it at home’ -> environment became less and less conducive to commerce and industry;
Greece/Rome - lack of institutions led to low social status of merchants (foreigners, slaves or freed slaves did that) Cicero: ‘of all sources of wealth, farming is the best’ ‘business on a small scale is despicable’ (but on large scale to a lesser extent)
Freed slaves also tried to turn into landed gentry -> so no class of entrepreneurs could form.</p>
<p>Lack of institutions -> state relied on private suppliers and subcontractors : provisioning of army and with construction contracts; mining and tax collection (all subcontracted to entrepreneurs - who took their share due to lack of oversight)
also lack of oversight led to arbitrariness of Publicanus. Livy: “where there was a Publicanus, there was no effective law and no freedom for the subject”; Diodorus: Spanish mines were very cruel due to this.
(? however: public contract were given only for five years ?)</p>
<p>still: Publiciani knights financed trade (but were excluded from senate, increasingly detested) - this had to be rich man with lots of land: in Rome only land could be offered as a security ! (in Greece that was different, so foreigners would take that role)</p>
<p>After first century a stronger imperial burocracy took over functions of Publicani; still instead of mutual control between senate and Publicani both colluded (oligarchic rule)
By the time of Septimus Severus regional armies would battle for power; no family stayed rich long - it took lot of money to pay off the soldiers of their own private army.
Commerce in third/fourth centuries was at standstill (lack of loans - no rich families left to provide securities + collapse of internal security)
Lots of taxes: the rich gained exemptions and pushed the burden to the lower classes; this ate away the basis of the economy.
Later times: much reduced economy centered around provisioning of army; less opportunity to pay for services with money -> non economic coercion becomes more important.</p>
<p>Economic reversal of end of Rome (beginning of feudalism)
- industry moved to self sufficient estates - to escape the militarized state and to save on transportation costs (commerce becomes uncertainty because of instabilities)
- everything made on spot - no need for trade; tie serfs to land - no migration.
- rulers debased/diluted coinage; also money went eastwards (all bad for commerce; but did strengthen tendency in direction of autarky)
—</p>
<p>Differences with modernity:</p>
<p>East:</p>
<ul>
<li>workshops were not autonomous - owned by temple/palace economy. All industry was self-financed; credit was for long distant trade only (?)</li>
<li>no land speculation based on rising prices ; property not bought on credit.</li>
<li>no banking intermediaries who would lend savings to entrepreneurs ; (commercial credit was granted among close circle of tamkaru merchants, at 20% interest - that was custom for many centuries; or Nadity/heiresses - they lent their inheritance and bought revenue yielding properties);
(Actually in OBE - tamkarum merchant (synonymous) were also active as money lenders, but they managed their own money, not deposits of others);</li>
</ul>
<p>West:
In Greece/Rome : lending/finances was activity of low esteem - delegated to foreigners or slaves.
In late Rome their were specialized investors into maritime trade (they had to know the details of shipping, from among retired captains)</p>
<ul>
<li>no intellectual property protection</li>
</ul>
<p>Finley (1973) says that in Rome and Greece (actually in OBE and Assyria things were more complicated)</p>
<ul>
<li>no productive loans</li>
<li>credit did not extend beyond available money; no credit creation/fractional banking (Greek law: purchase complete only if full price has been paid)
(probably this reflected the level of risk)</li>
<li>most loans were short term (for financing of a trading expedition)</li>
<li>Greek and Roman enterprises were organized as partnerships - each partner was liable for full amount of debt. This placed limit on size and financial complexity of enterprises. (Andreau 1999)
(joint stock companies with limited liability were not possible)</li>
</ul>
<p>‘the economic course of civilization has not always been uphill, as historians of technology seem to imply’</p>
<p>my notes on ‘Neo Babylonian Entrepreneurs’ by Cornelia Wunsch (source “here”:books.google.co.il/books?id=6rw1fxYalfUC&pg=PA62&lpg=PA62&dq=the+invention+of+the+enterprise&source=bl&ots=43aGWB_zdC&sig=eNXFZO6Ab_lorIDaveKEP1OM6C0&hl=iw&sa=X&ei=1Mg9VP-YAobbPZrRgagE&ved=0CGsQ6AEwCQ#v=onepage&q&f=false )</p>
<p>King often the largest land owner (other land held by members of court “land of the crown prince”; “land of the treasurer”)
New land along canals would be divided into parts(in return for military service)
Administration: oversee lands (land rent based on estimates of yield/harvest); collect fees for irrigation and rent; oversee and provision corvee service (for canal construction)</p>
<p>Canal construction -> royal task; watercourses, dams, locks, roads -> Temple task.</p>
<p>Temple lands:</p>
<ul>
<li>worked by un-free dependents (oblates - families were organized into plow teams)</li>
<li>worked by sharecroppers (share yield with temple)</li>
<li>later by rent farmers (had to pay fixed rent; had to invest into equipment for farming !, had to withstand risks - so these were entrepreneurs)
was this all a matter of forced participation ? Known cases where rent agreement was revoked due to insufficient inventories; some rent farmers had large debts -> implies these were large scale operations !</li>
</ul>
<p>After Persia conquered and imposed heavy taxes (-> intensivation of business)</p>
<ul>
<li>moved to cash crops or husbandry; cattle was scarce resource (had to be fed in summer)</li>
<li>temples outsourced more tasks to entrepreneurs.</li>
</ul>
<p>Prebends</p>
<ul>
<li>access to fishing & fowling controlled by licensing system</li>
<li>prebendary system: entitlements for temple revenue in return for services (requirement: cultic “purity” of holder + skills; later prebends could be sold - but only to peers)</li>
<li>prebends - similar to civil servants; (office was often hereditary, no risk involved ?, some functions were outsourced to commoners)</li>
<li>contrast to rentier prebends: entrepreneurs (did take risks) (still some prebend holders became entrepreneurs)</li>
</ul>
<p>Social status of entrepreneurs</p>
<ul>
<li>‘was socially rewarded’ no stigma of ‘dirty business’</li>
<li>some rent farmers were from prebend holder families</li>
<li>most were ‘social climbers’ - did not have a family name (meaning were not from the elite); often married into prebendary families - and tried to gain connections to royal administration.
(studied from archives of Egibi family)</li>
<li>Egibies invested in farmland and leased it to sharecroppers (provided benefits to tenants for planting date trees - cash crop)</li>
<li>opportunities: move to share crop/Niche products (like onions); transportation (to city) and org. of payment between tenant and temple/palace economy
(tax farming did that - tax collector gets a share but has to deliver quota; also did go between between tenant and institution;
for this you need a special relationship to the administration!)</li>
<li>also practice: the temple would advance payment (for seeds and irrigation); in term for debt obligations payed upon harvest (encourages market production - above subsistence level)</li>
<li>problem for producers: tax was payable in money (whereas transportation/distribution problems remain)</li>
<li>Egibi family operations (Neo babylonian empire + Persian empire (head of family had to make extended trips to Persia - to establish a special relationship with the Persian administration))
<ul>
<li>hire boats/boatmen to transport goods</li>
<li>supply landowners with money so they could pay for maintenance of canals (pay the temple)</li>
<li>debt contract with landowner: (lending / purchase and delivery agreement !)</li>
<li>toll/tax collection was part of the package (again advancing payment to landowners)
(? principle similar in scope to how Rome contracted out state functions; but more elaborate in operations + higher self risk ?)</li>
</ul>
</li>
</ul>
<p>Lending</p>
<ul>
<li>private lending to pay for taxes; to buy seeds; one could hire someone to do corvee or military service for oneself.</li>
<li>business lending: to increase liquidity, equipment, manpower, (seeds for farmer, or tax bridge) - for operations; (every loan had to be backed in full by securities)
no purchase of land on debt (-> no inflation of property prices based on speculation); could raise price of property only by improving it.</li>
<li>vertical integration: date producers may have dependent date beer brewers (etc)</li>
<li>partnerships (harranu): senior partner brings in capital (at more than the usual 20% annually !); gains divided 50-50 (Hammurabi); junior partners would often be ambitious younger sons (with limited means) and would try to pay back the lease (to be independent)
(Egibi’s started as junior partners; later they were the seniour partner in partnerships established for a business purpose (accounts were kept in archive)
complex transactions: Egibi’s borrowed to buy prestigious house near some high official; and then rented it out to their creditor.</li>
<li>over two generations the Egibi’s established relations with tax collection officials; third generation -> got till governor of Babylon.</li>
</ul>
<p>Inheritance
- Oldest son got at least half of the money (this prevented a dilution of the assets and preserved core business)
- concept of ‘undivided family’ - oldest brother represents all heirs collectively (this would delay split)
(unlike in Islamic law - here succession would often break up the business as is)</p>
<p>Slaves
partnerships could last for generations (unlike Rome where the partnership ended with the end of trading expedition)</p>
<ul>
<li>in neo-Babylonian times - a scarce commodity (price; several years income of a hired worker) (no slave stocked latifundia)
it made sense to raise the value of a slave by training !</li>
</ul>
<p>my notes on ‘Debt vs. Barter Theories of Money’s Origins’ by Michael Hudson
(Source “here”:http://cas.umkc.edu/econ/economics/faculty/wray/papers/hudson.pdf )</p>
<ul>
<li>
<p>payment for compensation of injury (most archaic); payment directly to victim or family (wergild);
a debt synomymous to guilt (Shuldt->Shulden;einloesen);
Greek timeE - ‘worth’, ‘esteem’ ; Homer’s usage ‘valuation of damage -> for compensation’ ; later (classical Greece) ‘valuation’ ‘wealth’; later ‘tax assessment’</p>
<p>Polyani: also had exchange by reciprocal gifts; also among family members loans were made without interest.</p>
</li>
<li>
<p>Palace/temple economy: redistributive mode, prices set by institutions.</p>
<p>The temple/palace often acted as creditor - advanced land/boats/workshops later respite on taxes; (also the origin of interest is probably here)
Silver was used as a measure of these debt obligations; (also to denominate debts between temple and palace)
‘Inasmuch as the major resource flows within the public institution were rations to feed their dependent labor, while the major payments from communities to the palace and temples consisted of crops, silver was made co-measurable with barley’
Weight standardization: (1 Shekel == weight of 240 barley grains in silver) = one bushel of barley (monthly consumption)</p>
<p>Temple/palace had its own land, cattle and dependent labour; developed account-keeping to plan activity; money as standard of value + mean of setting balances.
Temple in third millennium was sponsor of trade: ‘The fact that Enlil, the chief god of Nippur, bore the epithet “trader of the wide world,” and that his spouse was called “merchant of the world,” is an indication of the role of the Babylonian temples in the exchange of goods’</p>
</li>
<li>
<p>debt came before money (geld->gild (gothic: tax)); Oldest form: fixed obligatory payment to religious institutions (Mesopotamiam temple/palace economy); later: taxes on conquered cities.
payment was upon harvest.</p>
<p>was usage of silver simple, could it be used to ‘minimize transaction costs’ ? In OBE one shekel was a monthly salary => of limited value for consumer transactions but good for value store.
Problems: silver not uniform in quality (used 21karat); prone to manipulations -> need standardized weights + purity certification (need officials for both); many reports of transgressions.
These problems solved by standardized coinage only in 8th century BC !</p>
</li>
<li>
<p>free market exchange; price set by market; both market and gift economy co-existed with the palace/temple economy.
‘when crops failed late in the Ur III period c. 2100 BC, the price of grain supplied by independent producers rose sharply (Jacobsen 1953)’</p>
</li>
<li>
<p>market that sets prices came later;</p>
</li>
</ul>
<p>my notes on ‘The lost tradition of Biblical debt cancellations’ by Michael Hudson PhD
( Source “here”:http://michael-hudson.com/wp-content/uploads/2010/03/HudsonLostTradition.pdf )</p>
<p>Debt cancellations := policy designed to conserve the ‘self-sufficiency for the rural family-heads who made up the infantry as well as the productive base of Near Eastern economies’.
‘All these rulers seem to have recognized that if they permitted usury, debt-servitude and the sale of debt-slaves from one town to another to continue, much of the population would end up losing its lands and thus would be unable to pay duties or taxes, provide labor services or serve as a fighting force’
‘stopped emerging oligarchies from enriching themselves at the expense of the palace (and of social balance at large) was the fact that rulers repeatedly acted to subordinate mercantile wealth (especially usury claims) in the interest of promoting general freedom. The Middle Bronze Age was still far from being ripe for oligarchies to break free of palace control, to say nothing of unseating rulers.’</p>
<p>Also says that palace rulers prevented ‘emergence of powerful rivals’ by means of debt cancellations.</p>
<p>Argues that it is debt cancellation were act of ‘cosmic renewal’ of order (also usually took place at new year festival (in spring))
The ‘idea of unrestrained wealth-seeking’ was not accepted norm during pre-modern times. (?? isn’t that and idealization ; i would argue that these were compromises out of necessity: bronze age order was more fragile so that they had to compromise out of necessity ??)</p>
<p>Mesopotamian/Babylonain law codexes ‘were limited to the public sector in its interface with the rest of the economy. Laws such as those of Hammurapi were not a society-wide code but a set of laws governing public sector relations.’ (??? but it contains lots of criminal law elements too ???)
other cases (manslaughter,theft) were decided by oral common law (not by ‘eye for an eye’ !)</p>
<hr />
<p>Sumeria/Babylon</p>
<p>clean slate proclamations - amargi (also translated as liberty); canceled debts and freed debt slaves (but not slaves taken as prisoner of war). In Sumer these were Royal edicts, between 2400BC and 1400BC some thirty debt cancellations ;</p>
<p>Lagash: Enmetena in 2400BC - clean slate after big victory over Umma; Uruinimgina/Urukagina - as part of reform package (second year of office); also vows to protect ‘orphans and widows’
Gudea enacted clean slate on rebuilding of temple (start of reign)</p>
<p>Ur III (2112-2004 BC): Ur-Nammu and Shulgi - clean slate upon start of rule (usually Ur III is seen as ‘despotism with tighter military and bureaucratic control than Akkadian empire’)</p>
<p>Isin and Larsa (2000-1800BC) ; debt cancellation called andurar (freedom)</p>
<p>OBE: Hammurapi’s dynasty (1900-1600BC). debt cancellation = misharum declaration; Sin-mullait : inaugural debt cancellation (misharum) - to enable military campaigns: ‘ This gave them a stake in the society whose boundaries they were fighting to extend’. then repeated in 1803,16797 to ‘consolidate popular support’
Hammurapi - 1780 debt cancellation on eve of incursion to Tigris; 1771; 1763 - following victory over Rim-Sin. After Hammurapi : Ammisaduqa debt cancellation (1646BC) very detailed to forbid circumvention ! even undid land sales that were due to foreclosure on debt.</p>
<p>opposition to this practice becomes more sophisticated “The Danish cuneiformist Niels Lemche (1979:17) cites a document from the upstream town of Mari (ARM 8 33:13f.) dating from early in the eighteenth century BC when the city was ruled by one of Hammurapi’s contemporaries, Zimrilim. It stipulates that “this money shall not be released if a liberation should take place.” This meant “that a loan is not canceled in case an andurarum should be carried out.””</p>
<p>Practice of debt cancellation survived in Hurrian speaking north.</p>
<p>In Babylon: No debt cancellations in succeeding Kassite and then Elamite administrations; probably no centralized palace administration -> not many written records (dark ages). (however the Kassites kept other features of the old administration, like the Marduk cult)</p>
<p>‘Bronze Age rulers had pledged themselves to serve their local sun-gods by overseeing the rhythms of nature and society,’ these were replaced by ‘hereditary aristocracies and absentee ownership’
What happened was a ‘feudal-type privatization that spread from Mesopotamia through the rest of the Near East’.</p>
<p>Trade, land ownership + management of local temples were privatized -> increased monopolies lead to growth of Hapiru class (landless migrant labor) -> ‘new fugitives poured out of Mesopotamia’</p>
<p>“Whereas most debts in third-millennium Mesopotamia represented accruals of obligations to the palace and temples - and increasingly to their collectors “ - by first millennium debt was mostly to private owners.</p>
<p>Neo-assyrian empire: Andurarum debt cancellations seem to have happened by Sargon II etc: ‘its rulers consolidated their military strength by proclaiming andurarum to maintain a free (and hence, loyal) peasant army.’</p>
<hr />
<p>Israel</p>
<p>‘The Israelites are portrayed as having made a covenant to protect the economically weak by holding the land as the Lord’s gift to support a free rural population. “Land must not be sold in perpetuity, for the land belongs to me and you are only strangers and guests. You will allow a right of redemption on all your landed property,” and restore it to its customary cultivators every fifty years (Lev. 25:23-28). Israelite bondservants likewise were to go free periodically in the Jubilee Year, for they belonged ultimately to the Lord, not to any person (Lev. 25:54).’</p>
<p>In Israel these were sacred commandments “I, the Lord, am your God, who brought you out of the land of Egypt, out of the land of slavery” - to be free and economically self-supporting.
Also because of easier writing system (alphabet vs. cuneiform script) these laws were widely accessible (not restricted to the temple precinct as in Mesopotamia) Argues that following Bronze age collapse in 1200BC a ‘free for all approach’ developed that sanctioned private accumulation and resulted in ‘arrogance of wealth’</p>
<ul>
<li>it was not clear if these commandments were followed; discovery of Mesopotamian debt cancellations (during Bronze age) made that credible.
Nehemia upon returning from Babylonian captivity canceled debts; Shmita laws - ‘covenant with the Lord to promote economic justice in the land’.</li>
</ul>
<p>Dror - the word for economic freedom (similar to Andurar). ‘The story of Israel’s divine punishment served as a parable of how it would be rewarded for following a regime of economic justice but punished for permitting the wealthy to oppress the poor.’</p>
<p>800-600BC backlash of social prophets - the social protest against privatization of land + debt servitude that comes with it (Jeremiah)</p>
<p>Upon return from Babylon: Nehemiah & Ezrah proclaimed debt cancellation + tax remittance.</p>
<p>By first century CE ‘Rabbi Hillel developed the prosbul, by which borrowers signed away their rights under the Biblical laws. Hillel explained that credit would dry up without such a clause.’
“The fact that Hillel could establish the prosbul waiver as part of Jewish religion showed how far Israel had moved with the same tide of privatization that was sweeping the rest of antiquity into a new Dark Age.”</p>
<hr />
<p>Egypt</p>
<p>Rosetta stone is about debt remittance by Ptolemy V in 196BC; proclaims “to let everyone return to his home” -> Similar to Amargi/misharum.</p>
<p>Debt slaves not mentioned until new kingdom (1552-664BC) (prior slaves were all prisoners of war); Pharao Bocchoris (717-711BC) freed debt slaves. Described as way to return control to the crown (over private debtors). Later ‘palace power waned’ and the big privatization happened in Egypt too.? (Still Solon seems to have learned the idea from his travels to Egypt)</p>
<p>Alexandria Alfrano “Oriental Despotism and the Asiatic Mode or Production: A Modern day critique” (“Source”: https://kb.osu.edu/dspace/bitstream/handle/1811/45712/AlfaranoPaper.pdf?sequence=1 )</p>
<p>Marx: Oriental mode of production based on centralized despotism, bureaucratic administration, public ownership of land, large scale public works (via Corvee forced labor - this requires coercion); effective management is said to imply ‘supreme political power’.
Wittvogel: a complex system of irrigation can only be met by concerted effort of a great number of people, this created a manager class who would gain monopoly status (‘hydraulic monopoly’) and grab unrestrained absolute power, leading to permanent despotism; total submission is characteristic (not contracts)</p>
<p>MC Adams (1981): ‘uncertainties inherent in irrigation provided stimulus of growth of cities and hierarchically organized institutions ? to husband and redistribute subsistence and other resources’. Surrounding desert ecosystem added to fragility’ (other risks: crop disease, pests, flooding, salinization, incursions);</p>
<p>big temple/palace households also important as ‘reserve dietary fund’ - in times of famine; Big cities mitigated risks with their reserves - exchange mechanisms were more important; working against an irrigation only monopoly. Adams: construction and maintenance done by semi-independent contractors ; Rulers boasted of initiative in canal construction but these were later left in local hands.</p>
<p>Power seems to have developed due to question or organization + redistribution of surpluses (less over irrigation). Patriarchal groups led to larger households, and larger temples (temple structures are the first ‘big’ structures; large communities are pre-prerequisite to complex social organization)
First times of stratification in mid-third millennium (Feinman) ; however redistribution of surpluses happened on multiple levels (levels: palace and temple, traders were independent)</p>
<p>Upstream communities were better off in times of need (also were better off and developed hierarchies earlier) (? wasn’t Uruk further south, and the Euphrates too unruly in the north?)(in very early times the south would also get monsoon rains)</p>
<p>The result: surprisingly modern economy based on market mechanisms. (Marxist could not see that due to ideological constraints); Says that Marx/Wittvogel were ‘subconscious Orientalists’ (for lack of data and bias towards eastern cultures as being backward) (Wow).</p>
<p>my notes on: “A History of Sumer and Akkad” by L. W King</p>
<ul>
<li>climate of Egypt was better at preserving sites then (former) wet climate of Iraq.</li>
<li>abstract writing system of Sumerians presupposes extremely long period of previous development.</li>
<li>
<p>Sumerian - seems to have been a non semitic race … don’t know where they came from (maybe Persia driven by climate change, less likely from across the sea)
some suggest that the Sumerian language is a Indo-European language.</p>
</li>
<li>Sumerian script deciphered from Akkadian because the royal library of Ashurbanipal contained translation tables (for training of scribes)
long dispute if this was a different language/race (not semitic) because Akkadian and Sumerian have many things in common (Sprachenbund)</li>
<li>The rivers changed course to a great deal (moved to westward direction); Bablylon and Ur stand relatively near the Euphrates, but rest of them (like Sippar )is far away.
so canals had to supply the cities with water, ones the stream moved !</li>
<li>Almost all cities of Sumer and Akkad were on the Euphrates, not on the Tigris (??? what about Eshunna ???)
Euphrates has slower stream and has lower banks, it floods the country and can be used for reservoirs and canals.
Also Euhprat has high water levels till July (longer then Tigris); changed course frequently (generally moving westwards)</li>
<li>Akkad to the north of Sumeria; still adopted the culture/civilization of the south.</li>
<li>
<p>Names Sumer and Akkad where first used by kings of Ur III, they were the kings of ‘Sumer and Akkad’</p>
</li>
<li>main excavation sites: ancient city of Shirpula(Sumer name)/Lagash(later Semitic name) (excavated by M. de Sarzed de Tello)
City was not inhabited after it was sacked in early babylonian times; so the site lay undisturbed.</li>
<li>most important finds in area of palace and temple; the city was walled</li>
<li>main oponent of Lagash was city of Umma (Neo-Babylonian word) ; Fara/Shuruppak big site.</li>
<li>other sides have early strata from Sumerian period, but these cities were not abandoned: Nippur; Mukayyar/UR ;</li>
<li>
<p>Eridu was on edge of persian sea, but it moved; sandstone was used as bulding material (not bricks)</p>
</li>
<li>
<p>Semites had long beards; had prominent nose. (Semites during first Babylonian kinds shaved mustaches) ; full and fleshy nose and lips; wore loin cloths.</p>
</li>
<li>Sumerians shaved head and face (called the Semites - the black headed ones - ); had thick wool garments - petticoat; later mantle.</li>
<li>
<p>however Sumerian gods had long hair (maybe these were the gods of the land that the Sumerians ‘took over’ once they came in)
Sumerians at war: similar to closed Dorian phalanx. (also when they first came, they already had cuneiform writing system - even in early stage it already lost its pictorial form)
Maybe they were few in numbers and would still need Semitic neighbors and subjects; so that the two cultures influenced one another.
On the other side: early period the Sumerian god’s had shaved lips , in later times they had full mustache;
also the central god of Nippur was Enil; not the Semitic Bel.
Also early Sumerian text did not get any Semitic loan words - due to this lack of influence they infer that Sumerians came from the north.</p>
</li>
<li>they relied on genealogies of rulers taken from clay tables (often had their inconsistencies) (also they did not have radiological dating)
also used chronologies of major events written by the scribes.</li>
<li>in later babylon they counted years from ascension of the local king; earlier they counted years since noteworthy event such as building of temple ,
problem: events only of local importance; different systems of dating; so date-list had to be used to resolve conflicts, some of them lists survived.</li>
<li>
<p>also relative depth of artifact indicates its relative age; also style of art and style of writing indicate age; also if a solar eclipse is mentioned then its date can be computed.
(in Egyptology the chronology would be much simpler - the country was centralized most of the time).</p>
</li>
<li>
<p>art: early sculptures crude and in profile, later sculptures much more elaborate, full face.
style perfected under influence from Akkadia. Sumerian statue of ruler “Gudea”:http://en.wikipedia.org/wiki/Statues_of_Gudea (from Lagash) are of a style that has not been matched in later periods.
Also had figures cast out of copper; no two figures alike (the mold was broken to free the cast)
Author says that this was the high point of art; later Akakdians were more repetitious and less bold.</p>
</li>
<li>
<p>Sumerians used copper (Bronze came later - first at 3000BC ?) ; For Bronze they need both copper and tin, tin is a relatively rare metal that has to be brought in from far away, requiring long distance trading.</p>
</li>
<li>Cities: at first a collection of huts with a temple at the center; each with its local deity (later local deity (with local authority only) becomes part of pantheon).
this Probably happened as cities united into states: the main cities deity became the main deity.</li>
<li>the ruler Ensi/Patesi - rule of the city and representative of local deity. In Ur III the Ensi was a provincial ruler installed by the Lugal (king) or Ur (in early period Lugal was title of local king too - go figure)</li>
<li>all uncovered temples are of the later period (they were always extending it on top of earlier structures).</li>
<li>
<p>often the structures of the preceding period were used as foundations of structures build during later periods.</p>
</li>
<li>
<p>very early ruler of Kish: Utug (don’t know when he ruled); Utus deposited a vase into the temple of Enlil at Nippur (the religious center of Babylonia, cities sent offerings to Nippur on rating basis - under Shulgi); the vase deposited in commemoration of his victory over the land of Khamzi (as thanksgiving).
this implies that Enlil was recognized as a central deity in all of Sumer.</p>
</li>
<li>later: city of Kish ruled Sumer; under king Mesilim the Lagash and others owed allegiance. (hegemony passed frequently between the cities)</li>
<li>
<p>Lagash and Umma were neighbors (on different shores of a stream); Mesilim mediates. The resulting treaty does not name the rules,
it is formulated as treaty was reached between gods; the mediator being Enlil - the lord of the land (deity universaly recognized in Sumer)
Mesilim is mentioned, but only as having written down the contract; local rulers were perceived as ministers of the deity.
Stele was erected at the border as a symbol (first thing a subsequent raid by Umma did was to destroy it)</p>
</li>
<li>
<p>seems that Kish continued to rule after the death of Mesilim; there is a vase of later ruler Urzage found at Nippur;</p>
</li>
<li>Later Lagash (during reign of Eannatum) became stronger and took over from Kish. Lugal-shag-engur ruled Lagash comtemporary to Mesilim.
Badu; Ur-Nina - rebuilt the city walls of Lagash but did not conquer, no wars with Umma were recorded, his offerings mention that he built canals and temples, not as a warrior.
maintained relations with other cities by sending offerings: to Nippur; to city of Eridu and its deity Enki (prototype of Ea)
He built many temples in Lagash: to goddess Nina; the temple E-ninnnu dedicated to Ningirsru (god of battle)…. fetched/traded wood from mountains for construction of temples.
Canals: built a canal named Asukhur, seems to be a success - the economic basis of all this temple building.
Ur-Nina - pacific monarch devoted to welfare of his people and his city gods; prosperity lay the foundation for next rulers conquests.
Even the plaques depict him as laborer who is building ta temple or as carrying offerings (of course he is the largest figure)</li>
<li>The Plaques depict the king, next highest officials, next his sons (in order)
Successor of Ur-Nina was Arkugal, the fifth son; so they infer that there had been an interregnum and succession conflicts.
on other plates Arkugal had more prominent posture, so this theory is doubted.
(like photo of a central committee, position and posture of a figure is used to deduce his standing); still no minor conflict with Umma (nothing definitive)</li>
<li>Ennatum, son of Arkugal; don’t know if he maintained independence of Lagash, as they used Patesi title (however might not been too consistent in early period)
Lagash seems to have had allegiance to Akkad; renewed conflict with Umma
(the god of Umma warred with Ningirsu - rulers still only the officials of their city deity)</li>
<li>Ush was the ruller of Umma started incursions and plundering of the lands around Lagash; Ennatum (at the instruction of the city deity Ningirsu) fought back and sacked Umma).
All this is recorded on the steele of vultures - depicts vultures eating up the bones of the corpses of the defenders of Umma- as they were disposed outside the city.</li>
<li>peace was signed with Enakalli 0- the successor of Ush. A ditch/canal was dug as demarcation of the border (also some protection).
also both rulers would swear by their gods that the treaty would not be violated: the net of enlil would bind Umm if it were to violate the treaty, etc. etc.
(battle mace was also a weapon during this time)
a net scene is depicted on the stele of vultures - Ningirsu clubbing the caught enemy; the net here looks more like a cage…
‘the meshes of the net may be in a sense regarded as the words of the oath, by the utterance of which they have placed themselves within the power of the god whose name they invoked’</li>
<li>stele of vulture depicts Sumerians in battle: advancing in phalanx, leading rank protected by shields coverying the whole body (one shield carried by two).
lance bearers (standing in the gap between shields) also had an axe. When the enemy was broken the shield bearers would discard the shield and join in.
King wore a royal helmet that covers the whole head and neck. Shields and helmet made of leather.</li>
<li>also depicts burial scene of the war dead - were buried in mounds; elaborate ritual with offerings (of a bull?);
if they mention ‘heaped up burial mounds’ then that implies heavy losses.</li>
<li>last scene: Ennatum deciding on fate of POW’s.</li>
<li>also includes a list of other conquests of Ennatum (conquest of Kish, Erech and Ur and virtory over Opis and Elam) (most probably added at a later date); stella was probably in the temple of Ningirsu.
Elam is mentioned first because it was the hereditary enemy of all cities (Elam described as the ‘mountain that strikes terror’)</li>
<li>how comes Kish in here? probably was drawn into the conflict by supporting Umma against Lagash.</li>
<li>
<p>uses title of Lugal (king) after conquering nothen Kish; achieved hegemony over Sumer. Expansion probably in stages
stage 1) to the south; Ur, Erech Larsa Kesh
stage 2) Kish and Opis from the north want to check Lagash; Umma - one of the episodes of this war.
(chronology not always clear from the stele, different chronology in different records).
sites that offerings (as suzerain) were made to Ninkharsag of Kish; Enzu of Ur - all in the south, infers from this that conquest of south came before north.</p>
</li>
<li>
<p>warriors were drafted from people of Lagash, but campaings could not have been long. also people got something from grain tributes that other cities had topay.
also the lands of Lagash were not left un-tilled for lack of labor (otherwise prolonged sequence of campaigns could not have been).
area of cultivation was extended during Ennatum’s reign; also canals and reservoirs were build during later, more quiet period.
reservoirs would gather surplus water during spring, to feed the canals as constant source.</p>
</li>
<li>succeeded by Enannatum I - his brother. Umma was sem-independent but payed heavy tributes; as usual they rose upon the next succession.
(under Urlumma) tries to assert its independence by repeating raids on Lagash territory
these raids destroyed temples at the border stele - erected to symbolize prior treaties.
Umma tried to retain territory (unlike previous raids)</li>
<li>Records of Enannatum I say that his campaign was a victory; (his son Entemena records do not mention a victory, probably the result was not quite clear, as Umma was not sacked)</li>
<li>Ennantum I sent his people to Elam so that they brought back cedar wood for temple construction.</li>
<li>next ruler Entemena: again start of the reign marks war with Umma - a new ruler is a good opportunity to test his power.
unlike his predecessors he incorporated Umma into Lagash - and assigned his own official as ruler (not a native of Umma).</li>
<li>the priest Dudu was important - unusual inscription on a silver vase says ‘at the time that Dudu was priest of Ningirsu’ very unusual.</li>
<li>
<p>next ruler: Enannatum II - the last of the Ur-Nina dynasty.
next comes a series of short ruling rulers Enetarzi, Enlitarzi, and Lugal-anda(seems to have been a period of unrest)
Enlitarzi - the high priest during rule of Entemena is next ruler (!), is then suceeded by his (the former priests) son.</p>
<p>(Sign of instability: dispatch of priest of Ninmar to priest of Ningirsu: the priest of Ninmar led army to fight Elamites and divided the spoils with the local Patesi - means the he in turn was not fully in charge !)</p>
</li>
<li>
<p>Urukagina reestablished strong administration - and based his support upon the people (!) “in usurping the throne, he owed his success to a wide-spread feeling of discontent among the great body of the people.”
“His dropping of this time-honored designation may well have accompanied the abolition of privileges and abuses with which it had become associated in the mind of the people.”
When recounting prior history of struggle with Umma he only mentions them as victories of Ningirsu - no mention of his predecessors, a clear break.
also in first year of rule: discarded patron deity that old dynasty relied on for intercession with Ningirsu.</p>
</li>
<li>
<p>reforms of Urukagina: first describes abuses of the system: with conquests the rulers of Lagash changed a simple lifestyle (first among equals) to big palace and large courts.
taxes were increased and bureaucracy grew.’“Within the limits of the territory of Ningirsu,” says Urukagina, “ there were inspectors down to the sea.”’
the Patesi&Palace began to usurp position of temple and temple god - and began to be resented; and appropriated temple property. Priests began to grow fat and plunder.</p>
</li>
<li>So after a years rule Urukagina is dropping the title of Patesi in favor of King(Lugal). This transition might have seen actions to correct misuses.</li>
</ul>
<p>First stage of reforms: correct ‘isolated shortcomings’; ‘reform table’ enumerates problems that existed “since the beginning” :
* misappropriation of temple goods as pretext for reforms; restoration of old order where “patesi only received his throne as a trust to be administered in the interest of the god”
* Corruption from the top down, also the temple and crown - one hand was washing the other.
* Natural that with extended power of Lagash came an elaborate court, lots of temples and an army of officials:
“But with the growth of her power as a state, she lost many of the qualities by virtue of which her earlier successes were achieved. The simplicity, which characterized the patesi’s household at a time when he was little more than a chief among his fellows, was gradually exchanged for the elaborate organization of a powerful court.”
in less fortunate times the taxes to maintain all this were quite a strain…
“Urukagina records that his predecessors on the throne had appropriated the property of the temples for their own use. … The priests themselves grew rich at the expense of the temples, and plundered the people with impunity.”
* Palace began to usurp/overshadow role of the temple - appropriate temple property for their use; Palace priests would stele from commoners and divide spoils with palace.
Measures:<br />
* return palace land to temple.
* installed cult of goddess Bau
* proclaims that his own aim is to be administrator in the interest of the city god.</p>
<p>Second stage of reforms: “attack the abuses which existed among the secular officials and the priests”</p>
<ul>
<li>Sacks officials/inspectors, sacks officials who accepted bribes from</li>
<li>priests; high burial fees were cut down by half (? fees include a kid a bed and a seat ??); taxes/fees were cut</li>
<li>in ancient regime extortion and robbery were practiced by the powerful - they could take anything at will ! ‘sought to protect by law the humbler classes of his subjects from oppression by their wealthier and more powerful neighbors’
The powerful now had to buy things for a price, and the owner could refuse and may not be molested (rights of property !)</li>
<li>Fees for divination were abolished (Shekel was the currency !) (Former; one shekel for diviner, one for priest, one for Patesi - cut down to one shekel)</li>
<li>abolished divorce fees (this measure protected women, as payment of five shekel was enough to get rid of the wife).</li>
</ul>
<p>Laws of Urukagina are similar to laws of Hammurabi. Both drew legitimacy of laws on being given by the(pr local deity; both say said they were the champion of the poor against the strong.
(well slavery was maintained by both of them) “In his reign, he (Urukagina) says, to the widow and the orphan the strong man did no harm.”</p>
<p>However few references to other city-states (except for ties with Nippur and rebuilding of Enlil temple); still the city walls were rebuild.
clear that main interest were internal matters. Construction projects: temples, city walls;
(? “Wikipedia”:http://en.wikipedia.org/wiki/Urukagina says that the temple of Bau increased its dependent from 50 to 1500 - so some heavy redistribution seems to have taken place?)
“The host of officials he abolished or dispossessed of office had belonged to a military administration” … “When war threatened he must have found himself without an army and without the means of raising one.”</p>
<ul>
<li>Lagash was sacked by Umma (under Lugal-zaggisi); unlike previously this time they sacked the city; Lagash was later rebuilt, but as a state it never achieved its previous power.
(explanation for abrupt change: it seems that the elites - who were also the core of the army - were not ready to fight for the new order)
They found lamentations that grieved the fall - this is one of the first literary texts ever.</li>
</ul>
<hr />
<ul>
<li>
<p>for a time after the fall of Lagash: Sumer was governed by confederacy of states with Erech as capital.</p>
</li>
<li>
<p>Lugal-zaggissi then conquered all of Sumer and Akkad ; as written in offerings found at Nippur.
‘king of the land’ - ruler of all Sumer; later moved his government to Erech; Erech, Ur and Larsa - adjacent cities that formed the nucleus of the dominion.
(? these southern cities would be a problem for an dominion that stretches into the far north ?, maybe its all about a successful raid only)
ruled ‘lower sea to upper sea ‘ - may have raided as far as Syria (new policy! previously they did not venture that far)</p>
</li>
<li>seems to be that other rulers claimed to be ‘king of the land’ - from the same or similar time slice; go figure.
Lugal-kigubnidudu - another one from the same period.
Enshagkushanna - king of the land’; conqueror of “Kish the wicked” ; the north was probably under Semitic rule (mentions ruler “Enbi Ishtar” - Akkadic name)</li>
<li>Shar-Gani-sharri of Akkad has had a big empire.</li>
<li>King of Kish - Urumush - another semitic name; seems to have raided Elam (from inscriptions on his offerings sent to Nippur)</li>
<li>another ruler of Kish - Manishtusu : claims to have defeated the revolt of a confederacy of 32 city states;
obelisk of Manishtusu had inscriptions in Semitic Babylonian - records transaction to buy land and tree cities (!) and resettled the territory (= drove out the old tenants) + installed new managers from Akkad +
members of notable families from other cities (Lagash and Umma) !
policy of population transfer may have been due to political motives (one objective would be to weaken city of Akkad)
<ul>
<li>brought new workers to till the land. Another obelisk mentions vast expanses of his domain.
Process of centralization started under the kings of Kish;</li>
</ul>
</li>
<li>Also raided Elam - but also did not permanently occupy that country.</li>
</ul>
<hr />
<ul>
<li>the rise of Akkad and King Sargon (Shar-Gani-sharri)
Sargon was he called in later tradition (reference found in Ashur-bani-pal’s library)
Sharru-Gi-NA - was he called in Assyrian and Neobabylonian texts; now they found old stella with name of Sharry-GI- who may have been a predecessor. (no this is later ruled out: art work of his period much closer to Sumerian period, less similar to Akkadian style)
? did later periods mix it up and identify other kings as Sargon ? nw theythink that same descriptions are just similar forms of expressions describing similar events.
However events related to both Shar-Gani-sharri and Sargon are the same => the same one.</li>
<li>art from late rulers of Kish and Akkad is very similar; conventions based on prior Sumerian period; style of sculptures more varied and naturalistic.</li>
<li>Sargon was of humble birth; his father did not have any titles;</li>
<li>Sargon’s conquest of Syria was permanent - not just raids like his predecessors: position of capital made it possible to dispatch a quick punitive expedition if authority is questioned..
seems to have had all of Babylon + Syria and has crossed the Persian Gulf.
A system of regular communications - that’s what made the empire permanent ! (not just temporary raids now and then, communications maintained effective control)</li>
<li>
<p>deification: from now on rulers refer to them selfes as gods; practice may have been adopted from Syria (originated in Egypt).</p>
</li>
<li>An system of regular communications between cities: lots of records of commercial exchange: Akkad sent grain and dates; imported sheep and oxen.</li>
<li>Deportation/immigration from Akkad may have had objective of building influence in the provinces (similar to Manishtusu’s policies)
discontent over these policies might have caused revolt of all lands, during Sargon’s closing/late years.</li>
<li>
<p>revolts of all lands recorded during late reign (maybe backlash of policy of population transfer)
the (later) Chronicle says that a great famine in all of Mesopotamia destroyed Akkad (ascribes this to his evil deeds - just like in the Chronicles of the Old Testament)</p>
</li>
<li>Naram-Sin - son and successor of Sargon; siege of city of Apirak; expedition against land of Magan (not sure where that was); says he won nine battles in one year; conquests of Armanu (in Elam) and Lulubu (north to Elam)
calls himself “king of the four quarters” (of the world) - is this a claim of having had a world empire? - this title was no used by Sargon.</li>
<li>
<p>“Within the limits of Sumer and Akkad Naram-Sin appears to have followed his father’s policy of materially benefiting the provincial cities, while keeping their administration under his immediate control. “
rebuilt the temple of Enlil at Nippur and of Shamash at Sippar, temple of Ninni a Ninni-esh; votive offerings in Lagash.
Diarbekr stele - found in upper reaches of Tigris; (tells us that administration there was permanent, not just raid)
Stele of victory (in Tello/Lagash) - the enemy are Semitic, means that Sumerians were no longer present.
Culture of Sumer was taken over/transformed by their conquerors. sculptures are then free of rigid conventions;
Some of the rulers of Kish were deified; Naram Sin uses determinative for deity very often. Same cult for later rulers of Ur.</p>
</li>
<li>used archers to defeat Sumerians (Sumerians: spear bearers stand in phalanx formation; arrows can beat this up from distance)</li>
<li>Akkadians exerted strong influence upon Elam: here they adopted cuneiform writing and kept records in Akkadian language !</li>
</ul>
<hr />
<ul>
<li>artistic style of Akkad was the fusion of Sumerian and Semitic influences. “When the impulse was exhausted and the dynasties to which it had given rise had run their course, little further development along these lines took place.”
“Both in art and politics a Sumerian reaction followed the period of Semitic power”</li>
</ul>
<hr />
<p>In Lagash they found a long list of rulers from this time, some may have governed for only a few years; no mention of war/conquests and no use of ‘Lugal’ (only ‘Patesi’), might have been that Lagash was still under foreign domination.
Ur-Bau - one of this list has left records - did a lot of temple building; some</p>
<ul>
<li>Gudea - next ruler; does not mention geneaology (may be of obscure birth).
Very prolific temple builder (seven stage Ziggurat) - no Idea where he got resources for that (no wars/campaigns that could explain for it (except for one campaign against Elam).
Records vast operations for obtaining building materials; wood from Syria (means that Akkad was weakened); stones, copper, gold, etc. etc, got as far as the Mediterranean sea !
Trade between cities seems to have been unrestricted.</li>
<li>Seems to have coincided with progress in art of building: smaller bricks are used.</li>
<li>Big temple has two cylinders with inscription on building and decoration of temple: cause of building was after prolonged drought - so the temple was to atone for sins.</li>
<li>High point of art: big sculptures out of Diorite - very difficult to do that ; (also the stone were higher priced than silver!) ; later ones with a good sense of proportion.</li>
<li>Gudea built new canals and maintained the system of irrigation; his time was very prosperous; also big increase in amount of offerings.</li>
<li>Gudea upheld traditional privileges, such as the freedom from taxation ! (where did he get the money ??)</li>
<li>his “ideal of government was one of order, law, and justice, and the protection of the weak”.</li>
<li>during consecration of big temple “the maid was the equal of her mistress, and master and slave consorted together as friends … the rich man did not wrong the orphan, nor did the strong man oppress the widow.”</li>
<li>Rule of Gudea was regarded as golden age by later generation - he was deified by last king of Ur dynasty.</li>
<li>was this a peaceful time? Wiki says that emphasis on Ningirsu cult (god of war) would put that into question; also time following crash of Akkadia was full of conflicts.</li>
<li>Next ruler Ur-Ningirsu (son of Gudea); later he was high priest while the king or UR - Dungi ruled in Lagash ! (don’t know how Ur-Ningirsu was deposed); Lagash became a province of Ur.</li>
<li>Reasons for downfall? wealth from Elam introduced luxury and sloth; the empire surpressed local patriotism but did not create binding loyalty towards Ur; practice of divine rulers put them off too !
state was centralized but deification marks withdrawal of the king from administration.</li>
</ul>
<hr />
<p>After Akkad came the dynasty of Ur; political influence turned southwards as Akkad grew weak.</p>
<ul>
<li>
<p>Ur-Engur founder of dynasty of Ur
first he rebuilt temple of Nannar - the Moon-god (the famous Ziggurat of Ur), then city walls.
first conquest - city of Erech; rebuilt the local temple (of E-anna of Ninni) and installed his son as chief priest.
next conqured Larsa, then Lagash;
Installed his own folks as Patesi of Lagash; ruled over a confederation of city states.
rebuilt an Enlil temple at Nippur; expedition into Akkadian north, but no decisive victory.
did some expeditions into weakening Akkadian kingdom.
From now on: mostly uses Sumerian language (almost no Semitic inscriptions);</p>
</li>
<li>Next ruler Dungi(Shulgi): conquered the north too, unified Sumer and Akkad, ruled for 48 years (!)</li>
<li>reverses Akkadian tradition back to Sumerian roots: sacks Babylon; emphasises Eridu as center of worship; Only use of Sumerian.</li>
<li>adopts archery and the Akkadians way of war.</li>
<li>calls himself “king of the four quarters (of the world)” - like Naram-Sin of Akkad; rules over Elam - the source of great wealth</li>
<li>assumes divine status; renames seventh month in his name (first Sumerian to claim divine status)</li>
<li>empire of permanent status, like Akkad (not just raids) ; several expeditions to fight uprisings in Elam.</li>
<li>wiki: invested a lot to build roads; built rest-houses along the roads (the first inn)</li>
<li>commanded a large bureaucracy: couriers & messengers; collection + distribution of supplies; provisioning of slaves + oversee public works.</li>
<li>standardized weights and time keeping (official title of the year was fixed in Ur) throughout Babylonia !</li>
<li>
<p>tried to elevate the local deity Nannar - moon god; (like later Marduk - local deity of Babylon); but his dynasty was too short for that.</p>
</li>
<li>next ruler Bur-Sin; ruled only for nine years.</li>
<li>next ruler Gimil-Sin; ruled for seven years.</li>
<li>
<p>next ruler Ibi-Sin (last of dynasty) ruled longer - 25 years?. (all of them took divine status; continued policies of Gudea)</p>
</li>
<li>Omen tablet lists cause of downfall: probably an invasion from Elam. Ibi-Sin was captive and carried of to Anshan; (no confirmation from Elamite sources)</li>
<li>seems that Ibi-Sin gradually lost control of Elam, and that independent kingdom formed there.</li>
</ul>
<hr />
<ul>
<li>the empire survived the fall of the Ur dynasty; the change: the center of gravity was the city of Isin. (Sumerians too)</li>
<li>Isin dynasty lasted for 256 years (16 kings), mostly know about them from Nippur king list.(excavations came later then this book was written)</li>
<li>Ishbi-Ura’ founder; calls himself “King of Sumer and Akkad”</li>
<li>continued UR tradition of assuming divine rank.</li>
<li>Disintegration: (wiki) problem with access to water; also governor of Larsa - Gungunum seized power and seized UR, which was center of Gulf trade; this crippled Isin economically</li>
<li>? did end of Isis dynasty happened with the rise of Babylon? question was not settled at the time of this book (lengthy discussion follows)</li>
<li>end of Isis dynasty is end of Sumerian rule;</li>
<li>Wikipedia says that Sumerian “kings list”:http://en.wikipedia.org/wiki/Sumerian_King_List was drawn up by king of Isis; it had a political aim: legitimize Isin’s claims to hegemony when Isin was vying for dominance with Larsa.</li>
</ul>
<p>Next Babylon rules - center of power moves north. after waves of immigration Babylon became dominantly Semitic;</p>
<hr />
<p>Sumeria and Egypt;</p>
<p>assumes some influence upon Egypt ; early Egyptians used stone cylinder seals (later discontinued); like in Babylonia during all periods.
Other similarities: Slate carvings - same design; both used bricks for buildings; same wheat was planted in both places; Semitic element in Egyptian language was noted (similarity in names of gods Asuri->Marduk, Asar->Osiris); burial positions.
Was the Sumerian system of writing the parent of the Hieroglyphs?</p>
<ul>
<li>theory: semitic invasion of south Egypt during predynastic times; Semitic invasion was victorious (due to copper weapons) and the south conquered the north of Egypt (ascribed to Mena) the “blacksmiths” are said to have won battle after battle.</li>
<li>invasion theory would explain the break in culture between pre-dynastic and dynastic Egypt.</li>
<li>However more recent excavation put this theory in doubt, as there was more continuity between pre-dynastic egypt and later period.</li>
<li>early hieroglyps not that similar: only some self evident symbols sun - circle; mountains, sea; Cuneiform much more complex then early hieroglyphs - these were more pictorial.</li>
<li>
<p>also measured skulls of graves from predynastic and later times - and concluded that these are people of the same race (?) invasion theory was discarded.</p>
</li>
<li>form of bricks is different in both countries, may have evolved independently.</li>
<li>other influences such as crenelated form of building + round form of Gudea’s sculptures may have originated in Egypt and influence may have come via Syria.</li>
<li>
<p>still did not know how to explain other similarities: religion; presence of cylindrical seals + bulbous mace heads.</p>
</li>
<li>
<p>Elam. Akkadian invasion caused Elam to adapt Akkadian language and cuneiform scripts (even though thei had their own proto-elamite script). This might have been accompanied/or preceded by Semitic immigration;<br />
(strange picture: King of Elam writes in votive offerings to Babylonian deities in Akkadian language)
Later they used the same script to write in Elamite language.</p>
</li>
<li>Akkadian influence onto Crete/Minoan civilization; they did not invade Cyprus (that was a misunderstanding); May be that clay table/stylus (not the script) reached Cyprus.
Also in Knossos they have drainage systems that resemble those of Bablyon. Says its stange that Aegean culture has got so little influence from Akkadia via Syria.</li>
</ul>
<p>Notes on: ‘Ancient Mesopotamia / New Perspectives’ Jane R. McIntosh</p>
<ul>
<li>Mesopotamia: during ice age the sea levels were 100 meters lower; the whole gulf was dry; present levels reached by 4000-5000BC; 3000-4000BCE - water rises by 2 meters;
present water level reached between 1500-1000BC. ; higher Gulf means more extensive floods of the rivers -> more alluvial grounds.</li>
<li>Farming concentrated around slow moving Euphrates; it was easier to harness for irrigation. Settlements were abandoned when Euphrates changed course significantly.</li>
<li>Western desert: were impenetrable until domestic camel developed around 1000BCE;</li>
<li>Assyria - to north of Mesopotamia: 1000-3000BC more rain than today; easier for agriculture (today semi-arid); Taurus mountains with much wood (valuable resource).</li>
<li>
<p>Elam and Zagros foothils: marshes when the rivers reach the sea; agriculture possible near foothils, has five rivers; high ground is rich pasture.</p>
</li>
<li>diversion of rivers was frequent source of conflicts.</li>
<li>problems: erosion of mountains due to deforestation; irrigation cause salinization of soil (irrigation raises water table and raises salt; water goes, salt stays.
Second millennium BC: transition from wheat to barley (more salt tolerant) ; southern cities abandoned due to salt.</li>
</ul>
<hr />
<p>Respect for tradition:</p>
<ul>
<li>Ashurbanipal restored Susa temple (sacked much earlier by Uruk) ; created a great library in Ninveh and could read “obscure and confused inscriptionson stone from before the flood”</li>
<li>Nebuchadrezzar II restored temple at Ur;</li>
<li>Babylonian rulers liked to collect antiquities;
Other cultures</li>
<li>Israelites suffered from Assyrian and Neo-Babylonian expansionism - so the Bible portrays them as wicked and corrupt..
Also stories of delluge and tower of Babel are in the Bible.
Babylonian exile (588BC)</li>
<li>Greeks gave them a bad press (Herodotus); Ctesias - skewed account of Babylon via later Persian conquerors.
Berossus’s Babyloniaca - Bablyonian scholar writing in Greek (knew how to read cuneiform); Alexander the Great began reconstructing the ziggurat of Marduk (the Tower of Babel);
hanging gardens of Babylon were counted as a world wonder.</li>
<li>Europe: interest kept alive by the Bible; first travelers since the 17th century. 1807 - East India company creates outpost in Baghdad; C. J. Rich - first collector; 1825 - British Museum collection initiated.
1850ies - cuneiform script decoded, great interest and intense international competition at excavations.</li>
<li>interest: at a time when Darwin and Geology seemed to question the Bible, people found it reassuring that excavations yielded places mentioned in the Bible / confirmed authencitiy of the Bible.
Great excitement when George Smith found tablets in 1872 that mentioned the flood; 1870ies - greater care at excavations (prior to this they just plundered for antiquities)</li>
<li>Robert Koldewey perfected excavation techniques + studied sequence of construction (stratigraphy) now able to recognize and excavate mud brick buildings.<br />
Koldewey excavated the city of Babylon and found the Ishtar Gate (now in Pergamon museum) also excavated residential areas (new)
Walter Andrae used these techniques while excavating Assur.
(Wiki says that these techniques were first used in Egypt in 1880ies by Augustus Pitt Rivers and William Flinders Petrie http://en.wikipedia.org/wiki/Archaeology#Development_of_archaeological_method )</li>
<li>
<p>After WW I Britain was in charge of Iraq; permits were required for excavation, a portion of the finds was to stay in Iraw for the Iraq Museum;
archeology gets more professional; stratigraphic principles - differences in pottery used to assign finds to periods.
Main problem of chronology is mostly solved;
Leonard Wooley excavated at Ur. Discovered the royal cemetry - some graves with lots of objects; as if servants died with their masters. discovered thich layer without anything - said that this is evidence of the flood/dilluge.</p>
<p>Stratography became standardized in thirties (from early to later)
-Hasunna/Samara: (discovered by Seton Lloyd and Fuad Safar)
-at Halaf: pottery with beautiful decorations. (discoverd by Max Mallowan)
-Ubaid style pottery (discovered by Wooley)
-Uruk style: unpainted,
-Jemdet Nasr: geometric designs ;</p>
<p>Chicago team works in thirties (Jacobsen)
1930: excavation of Eshunna ; tablets were baked to make them sturdier; also location and depth was recorded for first time.
1933: aqueduct by Sennacherib discovered that was designed to bring “clean water to Nineveh from the distant mountains” (clues found in tablets)</p>
<p>Post WW II - Radio carbon dating in 1949; this helped to study the origins of farming and changes in the environment.
Mesopotamia was inhabited around 7500BC; 6500BC farming and pottery
sixties: archeology, studies contemporary population to shed light on ancient customs.
more sophisticated techniques: study of chemical composition of bones and pottery residues
landscape studies: settlement patterns, vegetation; 1956 Jacobsen says over irrigation and salination of soil responsible for shift of power northwards (2500BCE)
1948: Noah Kramer and Thorkild Jacobsen - scribal quarter at Nippur; tablets now backed + cast in latex to make a copy.
sixties onwards: rescue and conservation efforts.<br />
2003 onwards; everything breaks down due to bad security situation. Looting everywhere.</p>
<p>Languages:
at 1800BC Mesopotamian died out as spoken language; Old Akkadian - around 2000BC; split into Assyrian (north) Babylonian (south); 900BCE Aramaic in the west (replaced Akkadian as lingua Fanca)
Ideogram(Hieroglyphs) third millenium BC; cuneiform - syllabic script later in third millenium BC; was slightly adjusted for Akkadian; same script adapted for Elamite and Hittite.
11th century BC - Phoenicean alphabet - only took over when Persians took over Babylon.</p>
<p>Archeological methods of dating :</p>
<ul>
<li>dating by radiocarbon methods: problem: fluctuation of amount of C14 isotope in atmosphere; also imprecise for dates older then 1000BCE.</li>
<li>Dendrochronology - dating by analyzing patterns in tree rings allows for dating of timber - is used to calibrate radiocarbon method; still wood used in structures can be used only as an approx. date. also wood was reused as it was scarce;</li>
<li>Thermoluminescence dating: used for sediments or ceramics; energy from radiation decay is trapped by impurities in sediments; the older it is the more energy is absorbed. if heated to 5000C this trapped energy is released and measured. still imperfect method with 10% tolerance.</li>
<li>Dating by chronologies: list of kings (this one ruled for x years, etc); eclipse mentioned in text used as markers.
less certain in second millennium BC; remaining problems: gap between Akkad and Ur III. very early chronicles unreliable: kings were said to have ruled for centuries (can’t be);
King Ammi-saduqa of the first dynasty of Babylon recorded Venus table (first appearance of Venus in a year); three dates of accession would fit the curve (High (accession 1702BC), Middle chronology (accession 1646BC) and low chronology (accession 1582BC)</li>
</ul>
<p>Assyriologists like to rely on written records; later archeological methods would add to this knowledge, these are the only source for “finding out about the mass of the population generally ignored by the elite who ordered or produced the written sources.”</p>
<p>systematical and multidisciplinary studies do add a lot of clues:</p>
<ul>
<li>study of artistic representations adds detail -> knowledge of clothing and other aspects that have not survived; add data on available skills.</li>
<li>analysis of tools -> technological capabilities and how tools were used (from residues on them); this allows to infer practices.</li>
<li>composition of artifacts -> learn about patterns of communication and trade to get raw materials.</li>
<li>so they build a context of artifacts and use it to explain ritual practices and social organization.</li>
<li>modern ethnography adds how people would have lived in the past - given similar environmental constraints.</li>
<li>layout of houses -> learn about domestic organization; layout of public buildings -> clues about religion,politics and social practices.</li>
<li>regional survey studies: pattern of settlements / landscape; changes in course of rivers;</li>
<li>seeds and bones / remains of people -> what people ate, health & diseases; “patterns of birth and death, the gap between rich and poor, the activities performed by men and women”</li>
<li>nowadays they record position & context of tables -> system behind the records, relative importance .
sources need careful analysis of bias; if records and events are not from the same time then distortions / errors in copying etc. Context adds a lot to the understanding</li>
</ul>
</li>
</ul>
<hr />
<p>(History + politics)</p>
<ul>
<li>between 11500-9500BC - sedentary communities: settled permanently near nut bearing trees (? what about olives?) and cereals (some say first for feasting / bear?); this as climate settled after ice age (our climate at around 8000BC)
results in population growth; new tools and trading between groups - for what is not available locally.</li>
<li>colder an drier climate at 9000BC (?) and expanding of settlements into new areas forced them to plant seeds/agriculture starts (first in Israel and Nothern Zagros)</li>
<li>7000BC farming widespread in Mesopotamia; Pottery; packed mud houses (several rooms with storage room); leather clothing + linen textiles (?)_</li>
<li>6500-600BC Hasunah pottery style; self made, without ornaments; beginnings of irrigation; thirsty crops like flax required it.
(partially at the same time: Samarra pottery style - found in wide range of places!)</li>
<li>6000BC Halaf pottery style with geometric designs (thought to be done by specialist potters); some pots were traded over wide area (analysis of pots yields that)</li>
<li>5000BC Halaf and Ubaid people interact - and exchange pots. Ubaid style becomes dominant.</li>
</ul>
<p>Southern Mesopotamia;</p>
<ul>
<li>
<p>6200BC farming villages; earliest stage: building made of reed - did not leave traces ;<br />
Tell el-‘Oueili - mud brick buildings + Pottery similar to Samarra ; same culture ?
later: Ubaid1 period: dark painted pottery (few examples)
later: Ubaid2 period: more sites as farming expanded; boats used to communicate; six row barley planted - requires irrigation;
later: Ubaid3 (sixth-fifth millennium BC) ; pottery made with “slow wheel” tournette (first use; that’s why it replaced Halaf pottery) -> larger communities + specialization of crafts.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Ubaid4 same pottery in wider area ; south needs to trade to get copper; gold; timber from the north.
at Tepe Gawra - stamp seals with linear patterns; signs of administrative control;
Eridu - one of the first cities; tri party mud brick temples with offering tables - probably temples to Enki (god of waters and wisdom); o
most ordinary houses in Eridu,Ur,Uruk out of reed; further north they used mudbrick for houses.
at Eridu: family graves with some offerings; no graves singled out -> largely egalitarian society.
"Professor H. W. F. Saggs argues that Mesopotamian religious myths reflect the social setup at this time: Decisions were made by the whole community of gods in council together, and the voices of goddesses were equal to those of gods-a mirror of a society run by village assemblies, in which men and women had equal status" Also growing importance of temples, they are "the repository for its accumulated wealth"; also priests start to manage community affairs.
</code></pre></div> </div>
<p>Growing specialization: for pottery you need assistants to prepare the clay, etc. Copper foundry with forty workers found !
growing specialization + large number of workers -> need managers to ‘coordinate activities’.
management centers around large temples (with large warehouses) . 3300BC in Uruk at the Eanna temple - beginning of writing (we can’t read it);
cylinder seal appear; large number of bevel-rim food bowels - all of standard size (size of one daily ration for employees)</p>
<p>Similar development in Elam (Beveled-rim bowls, cylinder seals, wheel-made pottery, writing)
North remains fragmented and is less developed than Mesopotamia; still found trading outposts in the north.</p>
<p>Kings (might have been elected by assembly) and Priest (high priest en - partner of goddess in marriage); Kings acted in the name of the city deity
Uruk: Ianna - goddess of fertility; she made Enki drunk and stole the ME (attribute of civilization) - and brought it to Uruk.
Management: organized irrigation; controlled trading, trade expedition; production supervision; still they say that assembly existed !
at around 3000BC: Uruk at 60000 people !</p>
<p>4000-3100BC Uruk Period
Uruk culture: increase in aridity caused marshlands to dry out -> lots of fertile land fit for agriculture;
settlement density grew (exponential growth); specialization of crafts + social mechanisms to manage them: “proliferation of officially sanctioned individuals to administer them”
here they had the first urban landscape in the world.
Invented the ploughs and sledges; wheel invented - first for pottery then for transport.
‘“Secondary Products Revolution,” a quantum change in economic productivity and efficiency’
secondary products: Sheeps bred for wool; oxen used to carry ploughts (not just meat); cows, goats, milk used to do butter
Late Uruk - (from ap. 3100BC) - beginning of bronze age (copper finds + cuneiform scripts).
End might have been due to climate change: Piora oscilation http://en.wikipedia.org/wiki/Piora_Oscillation</p>
<p>“Here”:http://www.fofweb.com/History/HistRefMain.asp?iPin=ESCAW675&SID=2&DatabaseName=Ancient+and+Medieval+History+Online&InputText=%22social+organization+in+ancient+Mesopotamia%22&SearchStyle=&dTitle=social+organization+in+ancient+Mesopotamia&TabRecordType=All+Records&BioCountPass=0&SubCountPass=1&DocCountPass=0&ImgCountPass=0&MapCountPass=0&FedCountPass=&MedCountPass=0&NewsCountPass=0&RecPosition=1&AmericanData=&WomenData=&AFHCData=&IndianData=&WorldData=&AncientData=Set&GovernmentData= they say that society was quite egalitarian; however a permanent buerocracy existed.</p>
<p>Big changes: during third millennium kinship based community changed to hierarchical society controlled by temple and King.</p>
<p>3100-2900BC Jemdet Nasr Period; Proto-Cuneiform script (pictograhs);</p>
<p>2900BC-2334BC early dynasties (ED)
ED1: mythical times (kings list says rulers lived for century), depend on archeology to make sense of records.</p>
<p>2800-3000BC changes in south: big floods around 2900BC (thick layer of silt in Ur) Rivers consolidate into big stream -> needs lots of work to maintain irrigation.
2500BC - 80% live in the cities (?); cities tens of thousands strong.</p>
<p>ED2: 2500BC - evidence of wars over water rights; land, politics; temporary war leaders appointed by council turn into permanent Kings (though still governing in the name of city deity)
source of Gilgamesh epos - approx. 2600BC; Gilgamesh built Uruk city walls; war against Agga - king of Kish.</p>
<p>The temple administration dominates for most of the ED period (the kings only start to dominate at the end of ED period).</p>
<p>ED3: the dawn of recorded history. Seals of other cities found at sites, some say that “indicate the existence of cooperative leagues of cities.” (upon evidence that seals of city A found in city B)
cities would cooperate on irrigation projects and war; Royal cemetry of UR: increasing inequality with graves (rich graves with artefacts and tablets)
also raw materials indicate trading links to India and Afghanistan
First narrative texts that explain the action of Kings as being in accordance with divine will.
Mesanepada, king of Ur - also had title king of Kish (as did Mesalim - king of Lagash) ; don’t know if this means full conquest (+ submission of defeated party) or hegemony (+ allegiance of defeated party)</p>
<p>Nippur as center of spiritual authority - host of Enlil, don’t know when this started (some say ED1, some say ED3);
by end of ED3 Enlil regarded as chief deity of Sumer, maybe earliere Nippur was regarded as seat of council of gods; Priests of Nippur had a political role in all of Sumeria.
Eannatum of Lagash … “his victories nearer home may represent an early attempt to gain control over neighboring states rather than merely defeating them”
(more details on Lagash in the first book)</p>
<p>Akkadian empire (2334BC - 2000BC) (Semits take over); surprisingly few documents remain, what we know retold from later Ur III dynasty (with a skew to legitimize their own rule)
Sargon (Sharrum-Kin) - from humble background; ‘he was the son of a priestess and a man from the eastern mountains. After his birth, the priestess cast him adrift on the river in a reed basket; he was rescued by Aqqi, a water carrier, who raised him as a gardener. In his youth he served Ur-Zababa, king of Kish, whom he succeeded…’
Took over Mesopotamia from Lugalzegasi; got as far as Anatolia.
What kind or empire was his? Unified state in southern Mesopotamia (put in his own kind as local rulers, not leaving old establishment as vassals); extended influence due to control of trading route.
Benefits of unification: standardized system of weights, the same calendar; Akkadian as official language. Trade flourished -> income enabled large standing army (also had to put down frequent rebellions)
Daughter Enheduanna became priestess of moon god Nanna (first author known by name)
New capital - Agade - (location now lost)</p>
<p>another notable ruler: Naram-Sin (acession 2254BC. ) ; Claimed divine title (first); Empire disintegrated after Naram-Sin (due to Gutian invasions?)</p>
<p>Revival of Mesopotamia after downfall of Akkadia. Notable figures: Gudea of Lagash (known for constructions of temples) rich temples - indications that trade still flourished.</p>
<p>Ur III (third dynasty of Ur) 2112BC - 2004BC ; Ur-Nammu defeated Lagash and won hegemony. Son Shulgi focuses on internal matters; reorganized economy (strong state control); strengthened bureaucracy and weakened influence of temples - later installed standing army.
Shulgi proclaimed himself god (like earlier Akkadian rulers - so this may be a feature of strong state power) ; perform ceremonial marriage to godess ianna (formerly the high priest did that)
Propaganda: claimed continuity with first dynasty of Ur to strengthen their own legitimacy; as a measure recorded the Gilgamesh Epos; also wrote down Sumerian Kings list.
Control: Shulgi installed military governors (shagina) from his own country - his job was to control the local civil governor.
Governors had to pay rent for the land that they held (gun mada tax) (similar arrangement existed in Akkadian empire)
Shulgi had his law code - mostly it invoked financial penalties ! Large sumerian speaking bureaucracy; strict state control of foreign trade and industry.
State also maintained roads (with caravanserais) + royal messengers; canals; lots of scribes were needed, lots of accounting + new format for archives !
Decline: natural disasters; Amorite, Guti, Elamite attacks; inflation.</p>
<p>Following: no central power over Sumeria; Akkadian has replaced Sumerian (which is not spoken any more); some rulers tired to imitate the Ur III system; strong Elamite and Amorite influence.</p>
<p>Babylon: became important because the Euphrates moved westwards. First dynasty (1894BC) was Ammorite; Sin-muballit (father of Hammurabi) strengthened the cities defences
Hammurabi (1792BC ascension) ruled quietly for twenty years; then took over the region (seems to be a common pattern)
“Hammurabi was a strong ruler who took most of the business of government into his own hands, delegating little. His conquests coupled with major programs of land reclamation and irrigation gave him control of substantial lands,”
Successor Samsu-iluna had to fight rebellions. “Samsu-iluna’s tactics involved diverting the Euphrates to cut off Nippur. This spelled economic ruin in an area already suffering decline due to salination. Within eighty years of Hammurabi’s death, the state of Babylon was reduced…”
environmental pressures and raids of nomads lead to crisis of the whole system; Hittites sacked Bablon in 1595BC.</p>
<p>Kassite dynasty ruled for 576 years (quite successfull); Kassites were assimilated into Babylonian culture and worshiped Babylonean pantheon; at this time Marduk gains position of chief deity.
Later a Elamite dynasty took over;</p>
<p>North mesopotamia: large Hurrian state in 17th century BC; by 15th century this state - Mitani - is a big power (up to Levant, where they clashed with Egyptian interests); swallowed up Assyria.
Amarna letters - diplomatic correspondence (in Akkadian) ; marriages between rulers who called each other ‘brothers’; then Mitani state had problems with Hittites, that was when Assyria gained importance.</p>
<p>Assyria: king Ashur-uballit ; in the east took territories from receding Mitani state; in south Babylon.
Shalmaneser I defeated Mitani - now he was up against the Hittites; succeeding Assyrian rulers defeated the Kassite rulers of Babylon. Tukulti-Ninurta conquered Baylon but installed puppet king later (these were later deposed by restoration of Kassite rule)
relative prosperity from 1178-1133BC; decline from 1076-1030BC; and won big in Adad-nirari II won (911BC-891BC)</p>
<p>Bronze age collapse in 12th century BC: climate conditions caused famine; flow of streams was reduced until 950BC (Egyptians had their trouble with sea people)
Aramean tribes raided Mesopotamia from the west (Aramean became the new lingua franca)</p>
<p>Kassite rule collapsed in 1155BC; now second dynasty of Isin took over Babylon. Nebuchadrezzar I was defeating the Elamites and Assyrians.</p>
<p>Ashurnasirpal II won big time ‘washed his weapons in the Mediteranian’; replaced local rulers with Assyrian governors. Shalmaneser III -
aligned with Babylon - at end of reign fighting resumed, (King Jehu of Israel gave tribute and allied with Shlamaneser III)</p>
<p>Neo Assyrian empire (780-605BC): Tiglath-Pileser III (ruled 745-727BC) - of humble birth, ascended in the wake of civil wars.
reforms: professional army with Assyrian elite cavalry core, conquered people (Arameans) incorporated as foot soldiers; roads and messenger service.
Conquered the whole middle east.
Appointed eunuchs as governors (so they would not have their own dynastic ambitions)
Deportations , more detaails “here”:http://www.ucl.ac.uk/sargon/essentials/governors/massdeportation/</p>
<ul>
<li>Israel aligned with Egypt and revolted after the death of Tiglat-Pileser III; this revolt was put down resulting in deportation of the elite.
Sargon II (721-705 BC) of humble birth ‘He may have been the leader of a (successful) rebellion in Assur against Shalmaneser’s taxes’;
720BC peace with Babylon, expansions north and conflict with Egyptian interests; in 710BC conquered Babylon.
Sancherib (705-681 BC): large revolts after death of Sargon II (Judea again aligned with Egypt; Babylon; Elam) - put down by Sancherib.
rebuilt Ninveh, glorious palace ‘without a rival’; sacked Babylon in 689BC (after long war); this was sacrilege and unermined his authority.</li>
<li>Esarhaddon (681-669 BC) clemency of former enemies; rebuilt Babylon and encouraged worship of Marduk in Assyria. Defeated Egypt.</li>
<li>Ashurbanipal (669-627 BC); known for his famous library and benefactor of the arts; able administrator; conquered Egypt.</li>
<li>downfall (627-605BC); civil wars. Egypt conquered Judea (king Josiah);</li>
</ul>
<p>Neobabylonian empire (605-539BC) ; Nebuchadrezzar II fights Egypt and revolts in Israel/Levant; sack of temple in Jerusalem (589BC);
in later years he built a lot (Walls of babylon; ziggurat of Marduk; Ishtar gate; hanging gardes; Median wall)
Nabonidus; pious and ‘sought to uphold and revive ancient ways and practices’; also excavated ancient structures and built museums (!)
Worship of Sin + plague and famine (+inflation ?) made him unpopular. Cyrus exploited these sentiments in his campaign; then displayed clemency over the defeated;
continued to rule with the old administration + Marduk worship.</p>
<p>“Here”:http://www.fofweb.com/History/HistRefMain.asp?iPin=ESCAW675&SID=2&DatabaseName=Ancient+and+Medieval+History+Online&InputText=%22social+organization+in+ancient+Mesopotamia%22&SearchStyle=&dTitle=social+organization+in+ancient+Mesopotamia&TabRecordType=All+Records&BioCountPass=0&SubCountPass=1&DocCountPass=0&ImgCountPass=0&MapCountPass=0&FedCountPass=&MedCountPass=0&NewsCountPass=0&RecPosition=1&AmericanData=&WomenData=&AFHCData=&IndianData=&WorldData=&AncientData=Set&GovernmentData= it says that they were quite nasty to women and very militarized; “women were forced to wear garments that covered them entirely, not for the sake of modesty but to symbolize their utter lack of worth”</p>
</li>
</ul>
<hr />
<p>Economics</p>
<p>Mesopotamia lacks resources like timber, metals, salt, precious stones, etc; so it had to trade; Kings and temples created demand and sponsored trading expeditions
“Mesopotamian trade was never either wholly state-sponsored or completely private. Even under tight state control, merchants could undertake some trading on their own behalf, while private expeditions often had some state patronage and were subject to state taxes and regulations.”</p>
<p>most environmental constraints of the region did not change -> can learn from to days practices on how environmental constraints were tackled in distant past;
However: Nature changed (man made: deforestation, salination due to irrigation; environment: rivers change course, climate change) -> patterns of economic activity changed;
irrigation, barley and technological change enabled urbanization; however salination also brought ruin. Cuneiform records biased towards cities and public sphere, so much can be learned from excavated plants</p>
<p>Agricultural practice: sources “Farmers instruction” from 1700BC from Nippur;</p>
<ul>
<li>Spring equinox, rivers flood the land and spread silt; water softens ground and leaches away salt;
floods at harvest time! when it is not needed -> irrigation is a must. Canals need constant maintenance,supervision because of silt in streams.</li>
<li>when water recedes leases of land are negotiated (based on crop estimates), land now cleared with hoes and the wooden ard - simple type of plough.</li>
<li>Upon autumn rains: land ploughed and sowed (seeder plough - both ploth and seed dispenser, carried by oxen); high yields (1:10 - seed:crop ratio)
Ploughing was labor and resource intensive; fields long and thin - better water utilization + “to increase efficiency since a field’s optimum
length was the distance that a team could plough before requiring a rest, food, which were taken when the team was unhitched to allow the plough to be turned.”; Repeated surveying after sowing</li>
<li>after sowing the fields were flooded to top the furrow; again to remove salt; (then periodically flooded/irrigated for three/four times until harvest.’</li>
<li>harvest: barley in march (the main crop: tolerates salt and yields good); continued with veggies till june; harvesting by teams of three !
Then “the orphans, the widows and the destitute take their reed baskets and glean” (“The Debate between the Hoe and the Plough”)</li>
<li>
<p>threshing and storing of results in summer; land left fallow till next spring, grazing sheep and goats would fertilize with dung (?)</p>
</li>
<li>Crops were rotated: barley with peas (chick peas) (important to maintain nitrogen levels); crops had different water requirements;
Sesame - maybe important from Egypt via Syria (argue that Akkadian shamashshammu derives from Elblaite) ; maybe imported from India (argue that same oil (Sumerian ilu, Akkadian ellu) derives from Persian word)</li>
<li>veggies: Shade tree gardens, veggies growing in shade of date palms (Hanging gardens of Babylon)
Dates were important “Many surviving legal texts are concerned with the leasing and rental of date palms, estimates of their expected yield, their allocation to officials as part of their salary”
Onion: “Seeds were issued by the Onion Office to specialist growers, and the harvested onions were sent to favored individuals.” (2300BC at Nippur)</li>
</ul>
<p>Irrigation and flood control:
Clash over water rights were reason for war between cities; and conflicts between neighbors.
Kings were responsible for irrigation system (proud records of canal building etc); failure to attend to this duty could bring down a regime (?)
Peasants were legally responsible for maintaining adjacent canals; stiff punishments for failing these duties. (but they had fishing rights for their portion of the canal)</p>
<p>in school they learned how much water must be released to irrigate fields of given size;
Flood control: contain rising rivers and divert into reservoirs; canals with regulators to block/release water !</p>
<p>Excess water was ditched away (to prevent water level from rising, rising water brings up the salt); irrigation by flooding also removes salts.
“Ironically, since barley required only small volumes of irrigation water, its cultivation increased salinization” ?
Creation story Atrahasis; gods charged with canal works go on strike, so Enlil creates humanity to do the work instead.</p>
<p>Husbandry:
Households had one or two cows/oxen (for carrying ploughs, not for meat); only temples & kings had large herds;
Chickens introduced in 13th century BC from India !
Donkey as carrying animal; by UrIII Horse for chariots and elite. Camels around 1000BC;
The rich: Falconry a sport in 2nd millennium;</p>
<p>Nomads; in peaceful time co-existed with farmers - exchanged produce (would also be employed as shepherds for herds belonging to kings and temples); in bad times they raided;
Some rulers descended from Nomads: Assyrian king Shamshi-Adad and the Mari royal family; nomads were also traders (salt trade)</p>
<p>Meals: two meals a day; mostly barley meal cooked on a hot stone or in an oven-served with barley beer and a few vegetables.
Ordinary people had meat for festivals or marriages - would have fish and pork; bureaucrats and the elite ate mutton, ducks, geese, pigeons, and partridges.
Locusts were eaten (like in Egypt); Beer - consumed universally; “The daily ration issued to state employees was 1-4 liters”; the rich were drinking wine - chilled</p>
<p>Third millennium BC; Kings and Temples get large estates; early estates were regarded as property of city deity (King acted in the name of the deity);
later the kings grabbed land ownership; bureaucracy turned into “shepherds of the people”; Height of this centralization: Ur III king Shulgi (2094-2047 B.C.E.), “when the state had almost total control over the economy”
(also at that time the bureaucracy was recording everything; needed a large number of scribes)
later private enterprise gained importance</p>
<p>State revenues: taxation, tribute, rent; booty from warfare; offerings to temples (bala); merchants paid export taxes; nomads would pay for grazing rights;
conquered lands payed tax to the host; conquered lands had to pay taxes (gun mada) and could be claimed by the victor, who could install his people - in return for military service.
Victor could claim ownership of defeated Kings estate.
In Sumerian times land could be rented (for a quarter or half of annual yield) or bought (for a yearly yield) ; (sometimes complicated rules)
They had loans with interest - for financing of trading expeditions; the temples traded futures for incomes.</p>
<p>Population had to do corvee - forced labor (for canal construction etc) + military service in wars.
Industry: owned by temple, or kings palace ; private ; in UrIII there were textile works with 6000 people in it ! large number of professions.
Blin people were musicians or raised water for immigration (captured solders would be blinded);
Workers payed in rations (monthly pay or by the day), in first millennium BC - paid in silver shekels.</p>
<p>Units and measures: every city had its own; Akkadian empire and later UrIII standardize weights across the empire.</p>
<p>Trade: with Iran - across plateau (Gilgamesh epos: Enkidu expedition to mountains where he slays demon Huwawa whois the guardian of the cedar forest and ship the trees back to Uruk); around 2500BC across the gulf; Oman: copper; India: timer ivory gold
After sacking UrIII trade became smaller scale (less financial security); at around 1800 power shifted north; that was very bad for the gulf trade.
From 1750BC: coppery from Cyprus. Metals from Taurus mountains - the source of almost all silver.</p>
<p>Roads: Neo assyrians created network of ‘Royal roads’ with way stations; ‘Royal mail’ would swap horses here.</p>
<hr />
<p>Only few sites studied organization/layout of homes; palaces attracted more attention.
Area survey of Uruk, the first city: early period: 120-200 Hectars + 107 villages; ED1: 400 hectare + 4 satellite towns + 24 small towns + 140 villages; ED2-3: other cities in area compete with Uruk
After UrIII the area decline; Old Balyonia took over -> south was abandoned ! (partly due to change climate); 9th century BC: Assyria settled the area with POW’s; later neo babylonian revival;
In Assyria urbanization came later: majority of population was rural; fewer cities -> less competition between them.
General trend: strong state -> more resources for irrigation; more financial resources + security/stability; Collapse of state -> smaller economy.</p>
<p>Towns: unpaved roads; houses with raised thresholds (so that garbage is not washed into the house); Houses according to plan:
walled courtyard; rooms that face the courtyard: public room for guests; kitchen, bathroom (sometimes toilet); some houses with additional storie; in summer people slept on the roof.
Roof covered with wood beams (very valuable); sometimes room for workshop;</p>
<p>Near a city gate there was an open area - used for market and assemblies; garrison stationed nearby.
Outside the city walls: suburbs; also nomads/all visitors from other cities would be outside the city wall.
Furniture: shelves built into the walls ; wood table & chairs; wood chests;
Some houses had shrines; burial in the house floor (?);</p>
<p>Palace: was administrative,industrial, and economic center.
In Mesopotamia/Baylonia temple an palace were far apart (temple at city center); In Assyria temple and palace were close together, both surrounded by the same inner wall;
Cities in south more densely populate than in north; Hard to estimate size of city: census or tax records are missing (?!)
Palaces also organized around court yards; set of public rooms adjacent an outer courtyard; big throne room; inner court yards - adjacent to residential + service area; rooms often with decorations; .
Palace of Assyrian kings was very big.</p>
<p>typically >100 rooms; more than one store high.</p>
<p>Classes: who owned property (awilum); second rank - paupers (mushkennum), payed workers of the temple or palace estates, ‘their welfare was considered an important royal responsibility ; still could speak at assemblies; Slaves (wardum)
In laws: compensation for injury: highest for awilum; lower for mushkennum; lowest for wardum.
Scribes were both men and women; could be slaves (no high status associated with being a scribe)</p>
<p>Law codes: (from earliest) Uru-inim-gina of Lagash; king Shulgi of Ur III ; Lipit-Ishtar of Isin; Hammurabi. Early codes preferred compensation for punishments; Hamurabbi “an eye for an eye” seems to be influence of Ammoraites.
Law codes where not comprehensive (!) so in many court proceeding were based on local customs and traditions (when?)
Victim’s wish had weight; victim could ask for compensation or death as punishments for manslaughter.
Often ‘divine trial’ was used for conflicting opinions (jump into the river, if you sink then guilt has been established)
levels: local councils of elders/urban ward; Judge appointed by King ; the King as highest judge; Courts had to witness oral agreements (with witnesses)
Judiciary also enforced contracts between individuals (marriage as form of contract); enforce government decree and punish criminal behavior.</p>
<p>Prior to second millennium extended families lived in one house (larger houses); later only nuclear families (smaller houses).</p>
<p>High child mortality (she-demon Lamashtu was invoked to protect the small children); two-three children would grow up in a family; live expectancy around 70 years (?)
‘120 stated as the greatest age that the gods would allow’
Children (boys?) of the well-to do would go to school - ‘tablet houses’, payd by school fees (lots of beatings as punishment); writing leanred by copying texts;
later: both Sumerian (- dead language) and Akkadian was learned; teacher helped former students to get post in temple hierarchy.</p>
<p>Monogamy the norm; Girls married in their teens, boys in twenties - arranged marriages. If no children within three years -> ‘wife might then select a slave girl to act as surrogate mother for children that would officially be hers’ ; or second marriage.
‘A career as a prostitute was no bar to eventual marriage, although wisdom texts warned that ex prostitutes made inconveniently independent-minded wives’.</p>
<p>Men: role of patriarch; ‘Sons remained subordinate to their fathers even after marriage, and family property was controlled by the father until his death’;
Majority of population had to do with agriculture; Property relations & inheritance is where things get complicated …
had to do Corvee service (for canal construction) and take part in campaigns . In Assyria they had one year military service (ilku service)
On death property would be divided among sons (first son gets double share (sometimes)); then dowry for not maried daughters ; land would not be split (impractical) and the family continued to till in joint ownership (!)
Needed court approval to disinherit a son;</p>
<p>Women: wife (of well to do) confined to the house; ‘If things went badly, she and her children could be hired out to work, and in extreme financial trouble they could be pledged as slaves against unpaid debts.’
Poor mans wife worked in textile factory. Women could own property in their own right; could sue in court and act as witnesses; could have administrative powers - running estates in the name of the husband.
Widows would have same rights as men. If husband is missing the women could marry for a second time, but had to return if former husband reappeared.
Priestesses - daughters of well off; dowry they brought to the cloister would revert to family upon their death; in some places they were celibate (Nippur), others not (Sippar)</p>
<p>Slaves: early period: slaves are women, men POW’s are blinded to be kept as slaves (in most instances they were killed); Early Babylonia: slaves are important part of economy.
POW slaves property of the state (late Assyria, neo-Babylonian empire - soldiers keep slaves as private booty) ; in times of hardship (such as caused by debt) a man could sell himself (or other members of the family) into slavery - for limited period (several years).
Assyrian empire deported lots of people, from Sancherib onward they were royal slaves !
Temple slaves got same payment as free employees; ‘it was thought bad form to get rid of a long-serving slave’;
‘A slave girl who had borne a son to her master could not be sold, and she and her children became free on their master’s death’</p>
<hr />
<p>Politics:</p>
<p>Two centers of power: temple and palace; from Lagash records they learned that temple came first and that palace(secular) authority gradually became stronger at later time.
excavations in the seventies showed that they both temple and palace developed next to the other.
‘In the Sumerian city states the loyalties of the people and their rulers had long been to their city and its god’
Temple (+palace) redistributed surpluses; had a great number of dependent households (widows, orphans, POW’s; debt slaves)
Kinship groups (families) owned land of their own;</p>
<p>Bureaucracy; at Shurupak (2300BC-2600BC) they had units of hundreds, headed by department heads who where under head of state;
Head of state was En (in Uruk - he was the spouse of the city goddess); Sanga - title of chief temple administrator.
At some early time ‘the citizens may have played some part in selecting a king’s successor from among his sons or brothers’.
In early times there might have been some form of democracy: also is learned from ‘Gilgamesh epos’ and Babylonian ‘Epic of creation’; in both stories collective assemblies had decission roles(“here”:http://www.fofweb.com/History/HistRefMain.asp?iPin=MESP0167&SID=2&DatabaseName=Ancient+and+Medieval+History+Online&InputText=%22kingship+in+ancient+Mesopotamia%22&SearchStyle=&dTitle=kingship+in+ancient+Mesopotamia&TabRecordType=All+Records&BioCountPass=0&SubCountPass=1&DocCountPass=0&ImgCountPass=0&MapCountPass=0&FedCountPass=&MedCountPass=0&NewsCountPass=0&RecPosition=1&AmericanData=&WomenData=&AFHCData=&IndianData=&WorldData=&AncientData=Set&GovernmentData= )
‘Misfortunes reflected divine disapprobation, and an unsuccessful monarch could be overthrown’ Nippur priests could provide ‘divine’ legitimacy (that was important)
Might have been aliances between states in ED1-ED2; broke down by ED 3 (conflict of Umma and Lagash).
ED III - kings had territorial ambitions (see Lagash)</p>
<p>Empires: Akkadia Sargon and Naram sin institute provincial governors (ensi) + military governor;
From Akkadian times onwards - most of the time there is a political center. Role of temples is reduced - often appointed by the king (see Sargon’s daughter).
A very strict buerocracy has been created - governs and records everything - the model for the later Ur III state.</p>
<p>‘Ur III kings created an ideology that helped win them support throughout Babylonia, claiming descent from Uruk’s king Gilgamesh. They made much of their piety and role as divinely appointed protectors of the people and promoters of their well-being.’
maintained Nippur as center - instituted rotation system of offerings to maintain Nippur</p>
<p>Old Babylonian Empire - Temples gained in powers; private entrepreneurship flourished. Fusion of nomadic people and city states; first: states with ethnic basis (ethnicity came first - prior the city deity came first)</p>
<p>Hammurabi - centralized state again became stronger, private enterprise weaker. former free agents joined the state bureaucracy. Babylon became political, economical and spiritual center (Babylon with Marduk cult in replaced Nippur and Enlil cult); held position for a millennium.
Ilkum system - crown land granted (right of tenants?) in return for (military) service (later Assyria continued the same system)</p>
<p>Kassites took over but were assimilated into the system ; some commanders were granted land as recognition of bravery ‘Unlike a feudal system, moreover, the grant of land did not give the recipient control over the people living there.’
(longest ruling dynasty in Mesopotamia)
Mittani took over after Kassites - strict hierarchy. all land belonged to the king (temple had nothing to say). Vassals had to obey the king.</p>
<p>Assyrian empire (14th c BC); continued Babylonian ilkum system; officials held civil and military posts. transfer from landowning elite to crown - gained control of land; everything directly answerable to king ?
Governors installed fro among local elites who paid tribute to the center. Tiglat Pileser III - installs Assyrian governors everywhere (takeover upon rebellion)
Eunuchs preferred for governors (don’t have dynastic ambitions).
Temple integrated into power structure - acting as local judges; however temple subordinate to crown.
Lots of taxes: provinces were drained in favor or center; conquered people made up large part of army -> downfall.
Babylonia : special case; not administrated as province but treated as equal. Sancherib sacked Babylon, this was seen as impious hubris.</p>
<p>Local government -
villages governed by council; Town assemblies (consisted of all free males ?), some matters overseen by assembly of elders.
Big towns divided into wardens with own assembly - these local bodies dealt with divorces and inheritance disputes; law and order + sanitation; hearings on local matter.
tribal groups were under authority of local sheikh/elder.</p>
<p>Professional associations <br />
Trading quarters had their own offices outside the city - legally independent; would gather taxes, arbitrate in disputes; could call an assembly - to decide by vote.
In small town under Mittani : the mayor (hazzanu, sometimes known as rabianum) would collect taxes and was responsible for defense; would lease local property.</p>
<p>Propaganda
Role of a king :
external:
protect the realm (build walls, fight wars); ensure welfare by construction project for irrigation; promote trade (security along the roads; fix weights and measures, maintain roads)
internal:
Justice and care for widows and orphans (some kings canceled debts (misharum) !);
Propaganda: grew more detailed; ED times - dedications of offerings or temples; UR III - the kings list, Gilgamesh Epos; Neo-Assyrian empire: detailed description of campaigns in inscriptions/reliefs</p>
<p>‘Success was in itself indicative of legitimacy and divine approval.’ no divine approval can lead to replacement
Usurpation would be justified by misdeeds of the predecessor.
Lots of things would serve propaganda: city walls, imposing temple/palace structures; inscriptions/stele; records of past rulers (to support the preset lineage).</p>
<p>Importance of trade (due to lack of raw material); traders had important role in diplomacy.
Envoys carried messengers (were often illiterate); sometimes were
Protocol for exchange of messages: sending party has to provide escorts for messenger (lack of escort is a grave offense)
treaties were ratified by oaths; (invoked divine retribution for breach); sometimes sealed by royal marriage; sometimes prince hostages would be taken as securities.</p>
<ul>
<li>Education system seems to have been based around copying texts, that was the way of learning; in post sumerian times Scribes learned both Akkadian and Sumerian.
The height of literacy and literary accomplishments seems to have been the Old Babylonian empire (most Sumerian texts survived as copies from that period)</li>
</ul>
Finnland's schools
2014-07-08T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2014/07/08/finnland-school
["", "blogg"] | <notextile>Finnland's schools</notextile>
<hr />
<p class="publish_date">
04 May 2013
</p>
<p>Phil Greenspun reviews the book <a href="http://www.amazon.com/gp/product/B0061NT61Y/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=B0061NT61Y&linkCode=as2&tag=pgreenspun-20&linkId=VSNDYSKRJD72L4ZD">The Smartest Kids in the World: And How They Got That Way</a> by Amanda Ripley, his review is <a href="http://blogs.law.harvard.edu/philg/2014/06/25/smartest-kids-in-the-world-finland">here</a>
; it is full of praise for Finland’s schools - the <a href="http://en.wikipedia.org/wiki/Programme_for_International_Student_Assessment">PISA</a> test compares student performance between countries and here Finland used to be <a href="http://en.wikipedia.org/wiki/Programme_for_International_Student_Assessment#Results">very good</a> .</p>
<p>The secret sauce is said to be that teacher training colleges are very selective and that they only accepts the top 20% of applicants. Teachers are well respected and in good pay. Teachers in Finland are not micro-managed and do not have to follow a detailed curriculum.</p>
<p>My question is this one: how do they educate teachers so that they are able to work independently, without the usual micro management? I know of some engineers who are not very independent thinkers, despite the good grades that they must have had. So it must be more then just the selection process.</p>
<p>Maybe its time to learn Finnish …</p>
<hr />
<p>Some more details on Finland’s school reforms are in this article:</p>
<p><a href="https://edpolicy.stanford.edu/sites/default/files/publications/secret-finland%E2%80%99s-success-educating-teachers.pdf">The Secret to Finland’s Success: Educating Teachers</a> by Pasi Sahlberg</p>
<p>Here it says that even Finnish basic and high school teachers must have an MA degree; preschool and kindergarten teachers can do with a BA; An aspiring teacher is supposed to be trained in doing research on his own; so they do teach them to be independent.</p>
<p>Really impressive. I wonder who was the person who pushed for such massive change, and how did he/she succeed to persuade decision makers.</p>
<hr />
<p>However Prof. <a href="http://en.wikipedia.org/wiki/Finnish_Civil_War">Hannu Simola</a> says that there is no secret sauce, also this is all not quite applicable to the rest of the world.</p>
<p><a href="http://matex.zozlak.org/DataBases/PISA%202000-2009/Simola_2005_Finnish%20PISA%20Miracle.pdf">The Finnish miracle of PISA: historical and sociological remarks on teaching and teacher education</a></p>
<ul>
<li>In Finnish society they have a strange mix of authoritarianism and collectivism; attitudes seem to be similar to those in Russia - well Finland was part of the Russian Empire from 1809 until 1917. Finland has a history of an ‘agrarian and pre-industrialized society, the ethos of obedience and subjection that may be at its strongest in Finland among late modern European societies’
So there is just no problem with maintaining discipline in class. I guess discipline is also easier to maintain because Finland does not have any big cities with a multi million population; the largest city is Helsinky with <a href="http://en.wikipedia.org/wiki/Template:Largest_cities_of_Finland">one and a half million</a> , it is the only city with more than a million people in it.</li>
</ul>
<!-- -->
<ul>
<li>Finish teachers were always conservative and identified with the upper middle class (almost no leftist teachers even during the <a href="http://en.wikipedia.org/wiki/Finnish_Civil_War">Finish civil war</a> )
Professionally they are also very conservative: “Finnish teachers spoke to their pupils mostly as adult models and keepers of order and safety in the classroom. Rather than encouraging intimacy, some experienced Finnish teachers emphasized how important it was to keep a certain professional distance from their pupils and their homes and problems”. Unlike in Sweden, most Finish teachers are said to be standing in front of the class and are strictly following the textbook.</li>
<li>The reforms of the seventies: unified the school system; raised the entry bar to the profession + education requirements for the teacher profession (must have an MA degree).
The side effect of raising the status of teachers: “people have been awakened to the fact that it is only through education that it is possible to climb the social ladder, or even to keep up one’s position.”</li>
<li>Reforms of the nineties: no individual teacher assessment, but the results are measured; Therefore teachers are stressed as they are measured by results, not by compliance with instructions.
The ‘paradox is that the politically and pedagogically progressive comprehensive school reform is apparently being implemented in Finland by politically and pedagogically rather conservative teachers’</li>
<li>as a whole Finnish teachers are still very satisfied with their job.</li>
</ul>
<hr />
<p><a href="http://www.mv.helsinki.fi/home/hsimola/">Prof. Hannu Simola’s</a> research interests are now in the area of ‘effects of quality assurance and evaluation (QAE) as a new technology of governance in education and educational policy transfer’ . Very interesting topics, unfortunately most of his research is out of reach for me; still this is all very interesting.</p>
<hr />
<p>The lesson here is that each society gets the school that it deserves; A very good school system has a positive influence back upon society, in this event we would get a positive feedback loop. An example of this feedback: Finnish society respects teachers and therefore education as a whole is receiving a higher standing.</p>
<p>Another lesson is that the <a href="http://en.wikipedia.org/wiki/Programme_for_International_Student_Assessment">PISA</a> evaluation does apply objective criteria to measure student performance; Because of this it seems to be favoring schools in societies that enforce strict norms of obedience; Sort of: “Here is the stuff you are supposed to learn by the end of the grade, now please be so kind to toe the line”.</p>
<p>As a measure of performance PISA does not measure such things as intellectual independence or ability to work in groups. Prof. Simola seems to be quite fond of the Swedish effort here. In the more individualistic society of Sweden they seem to have put a lot of effort into this direction; these are the traits valued by Swedish society, so they train their kids accordingly.</p>
<iframe width="420" height="315" src="https://www.youtube.com/embed/hhXpGRJQV4Y" frameborder="0" allowfullscreen="">
</iframe>
<p>In the words of Jethro Tull:</p>
<p>“Cause you were bred for humanity and sold to society —
one day you’ll wake up in the Present Day —
a million generations removed from expectations
of being who you really want to be.”</p>
<p>all together now:</p>
<p>“skating way on the thin ice of a new daaaaaaay”.</p>
<p>The problem is that ‘creative thinking’ and ‘ability to work in groups’ are categories that are impossible to measure objectively; if an education system focuses on such things then it becomes impossible to measure its performance in any meaningful way.</p>
<p>By focusing on objective criteria the Finns seem to be giving their education system a good shake up, of course it serves them well to have well trained teachers to address this challenge; I guess without this stimulus it would have been easy for Finish teachers to become complacent and inefficient.</p>
Image archive organizer
2014-06-17T00:00:00+00:00
http://mosermichael.github.com//all/projects/2014/06/17/imgarchive
["", "projects"] | <h2 id="image-archive-organizer">Image archive organizer</h2>
<p class="publish_date">
24 February 2014
</p>
<h3 id="the-problem">The problem</h3>
<p>Lots and lots of images everywhere … in the course of a few years we have managed to accumulate thousands of images from various cellphones and digital cameras, and we never had the time to organize a proper photo album.</p>
<p>It all looks like this <a href="http://xkcd.com/1360/">XKCD comics</a>
<img title="Wow, ANIMORPHS-NOVEL.RTF? Just gonna, uh, go through and delete that from all my archives real quick." src="http://imgs.xkcd.com/comics/old_files.png" /></img></p>
<p>So its time to write a program that makes some sense out of this mess …</p>
<h3 id="defining-the-program">Defining the program</h3>
<p>The program should be able to maintain a picture archive; here are my requirements:</p>
<ul>
<li>The program organizes a picture archive; images are stored as files, the program also maintains some data on its own such a as a list of files, etc.</li>
<li>It should be able to exclude identical images; if you import two identical images twice then it should be able to skip copies.</li>
<li>It is very hard to work with thousands of image files all in the same directory (tried that); the program should be able to organize images by chronological order, within the archive we end up with a hierarchy of directories <year that="" image="" was="" shot="">/<month image="" was="" shot="">/<day image="" was="" shot=""> ; Image meta data includes the creation date ([exif format](http://en.wikipedia.org/wiki/Exchangeable_image_file_format) ), so the program should not rely on the file date alone.</day></month></year></li>
<li>Checking the archive: the program should be able to check the consistency of an archive. If a file has been corrupted then it must be thrown out of the archive.</li>
<li>Merging of two archives ; if you have two picture archives then it should be able to compare both of them and to add missing information from the other archive; all this would help with creating back ups of the archive.</li>
<li>The program should have a command line interface; good enough for me (but not enough to make money out of it ;-)</li>
</ul>
<p>This program can be of use in the following scenarios</p>
<ul>
<li>if you want to sort out a big stack of pictures then the first thing would be to organize them by date; you can begin to filter out the good pictures once each picture set is of manageable size, then you can decide on what to upload to picassa or to print on paper.</li>
<li>People like me who want to have their pictures stored locally on their own systems, not in some distant Cloud ™; I think end users are increasingly reluctant to trust the cloud with their own personal data - i personally see the cloud thing as a huge centralization of computing that inevitably comes with centralized political control (more on that <a href="http://mosermichael.github.io/cstuff/all/ramblings/2013/06/08/post-1.html">here</a> )</li>
</ul>
<h3 id="how-to-use-the-program">How to use the program</h3>
<h4 id="creating-an-archive-for-the-first-time-or-adding-of-pictures-to-an-existing-archive">Creating an archive for the first time or adding of pictures to an existing archive.</h4>
<p>A short description of that functionality:</p>
<pre>
img-archive.pl -i -s <dir> [ -s <dir2> ] -d <dir> [ --noprogress ]
img-archive.pl --import --source <dir> [ --source <dir2> ] --dest <dir>
import images from source directory (specified by -s <dir>) to archive (specified by -d <dir>)
Multiple source directories can be specified by repeating option -s
Images are ordered into a directory hierarchy
<year the picture was taken</>month picture was taken</<day picture was taken>/<file name>
The date of picture taken is extracted from the image file (where possible) -
this is enabled by the ExifTool, written by Phil Harvey.
The script checks if the picture already exists in the destination directory;
The option --noprogress suppresses display of progress indicator during copying.
Script is part of my simple picture archive system.
</pre>
<h4 id="backup--merging-to-archives">Backup / Merging to archives</h4>
<p>A short description of that functionality:</p>
<pre>
img-archive.pl -m -s <dir> [ -s <dir2> ] [ --noprogress ]
img-archive.pl --merge --source <dir> [ --source <dir2> ]
Merge a number of archives; after merging every archive should have the same set of data;
However the file names may differ.
</pre>
<h4 id="verifying-the-validity-of-an-archive">Verifying the validity of an archive</h4>
<p>A short description of that functionality:</p>
<pre>
img-archive.pl -c -s <dir> [ -s <dir2> ] [ --noprogress ]
img-archive.pl --check --source <dir> [ --source <dir2> ] [ --noprogress ]
checks an existing archive for consistency;
</pre>
<h3 id="how-to-install-it">How to install it</h3>
<p>The program is written as a perl script (caugh caugh). It can be used on both Linux and Windows;</p>
<p>So on Linux you need</p>
<ul>
<li>
<p>the <a href="https://raw.githubusercontent.com/MoserMichael/cstuff/master/img-archive/img-archive.pl">img-archive.pl</a> script</p>
</li>
<li>
<p>perl installed</p>
</li>
<li>
<p>perl package Exif, written by Phil Harvey. (<a href="http://cpansearch.perl.org/src/EXIFTOOL/Image-ExifTool-9.60/html/ExifTool.html">here</a> and <a href="http://www.sno.phy.queensu.ca/~phil/exiftool/ExifTool.html">here</a></p>
</li>
<li>jpeginfo program written by Timo Kokonnen (<a href="https://github.com/tjko/jpeginfo">here</a> )</li>
<li>openssl must be installed (comes with most distros default installation)</li>
</ul>
<p>To simplify things you can just download and unzip <a href="https://github.com/MoserMichael/cstuff/raw/master/img-archive/img-archive.zip">this archive</a> The archive includes:</p>
<ul>
<li>
<p>the program</p>
</li>
<li>perl package Exif</li>
<li>jpeginfo program compiled on windows for Cygwin (if that stops you can just build it by going to jpeginfo_prog directory and run make).</li>
</ul>
<p>You still need an installation of Cygwin with perl and openssl.</p>
<h3 id="a-more-detailed-description-of-the-program">A more detailed description of the program</h3>
<p>For each image that is part of the archive we first compute the SHA1 signature/checksum of the image data - the digest is a function that takes the image data as input and that produces a large number as output; one might wonder if it will ever happen that two different pictures will yield the same signature value, however the likelyhood of such an event is considered to be very very low (<a href="http://en.wikipedia.org/wiki/Data_deduplication#Drawbacks_and_concerns">see here</a> ) , surprise, surprise.</p>
<p>Each archive has an index file hash2file.lst - this file lists both the file name and its digest/hash number. First thing when working with an archive is to open the index file;
For each entry in the index file it checks if the file exists at all if the file does not exist then the index entry is discarded; In memory a hash map is mapping the digest to the file name; this hash map is used to check for duplicate images;</p>
<p>The different commands in more details:</p>
<h4 id="import-of-new-images">Import of new images</h4>
<ul>
<li>The index file of the destination archive is read; checks if all files that are listed in the index file do exist; for each index entry a mapping is established in digest_to_file (maps digest to file name) and has_file (maps file name to constant 1)</li>
<li>All files in directory of destination archive (other than the index file) are listed; if no checksum for this file exists, then the checksum is computed and an entry is added to the index.</li>
<li>All files in the source directories are listed; for each file the SHA1 digest is computed, if the digest does not appear in the archive index (no entry in digest_to_file_exists) then the file is imported;</li>
<li>Import of a file
<ul>
<li>Extract Exif meta data from the image file; get the CreationDate (if not available DateTimeOriginal, if not available then ModifyDate is extracted; if that is not available then the OS file creation time is assumed.</li>
<li>Target directory is formed as <Year>/<Month>/<Day> under the target directory</Day></Month></Year></li>
<li>File name is that of the source file; take care not to overwrite an existing file, if a file of that name exists then pick a different one.</li>
</ul>
</li>
<li>Save the modified index</li>
</ul>
<h4 id="merging-of-archives">Merging of archives</h4>
<ul>
<li>Check that all directories are valid archives (i.e. that each archive contains an index file) ; read in the index for each archive;</li>
<li>For each archive; enumerate the entries of the index; check if a file of the same digest exists in the other archives, if not then import the image into the respective archive</li>
<li>save the modified indexes.</li>
</ul>
<h4 id="integrity-check">Integrity check</h4>
<ul>
<li>The index file of the archive is read; checks if all files that are listed in the index file do exist; for each index entry a mapping is established in digest_to_file (maps digest to file name) and has_file (maps file name to constant 1)</li>
<li>All files in directory of destination archive (other than the index file) are listed; if no checksum for this file exists, then the checksum is computed and an entry is added to the index.</li>
<li>For all entries of the index: compute the digest again, check if the digest has not been changed;</li>
<li>If the digest has been modified, then check if the image file is still valid; if it is valid then modify the existing index entry, for invalid files delete the index entry</li>
<li>Report Number of valid, number of modified and number of invalid files;</li>
<li>Save the modified index.</li>
</ul>
<h3 id="an-ode-to-perl">An ode to Perl</h3>
<p>Perl is like an old friend that is always there for you; whenever there is a problem it comes in with a huge set of libraries that seem to cover almost all areas, whenever there is a problem one can easily make up a program to automate the process. For example during the recession of 2001 it helped me to make up a program to spam prospective employees with my CV, the script would parse job boards for adds, build a database of adds and send out the CV; somehow that helped me and some of my friend;
Things like that create brand loyalty, at least for me ;-)</p>
<p>Perl the duct tape of more than the early internet !</p>
Trouble in the ex-USSR
2014-05-20T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2014/05/20/russia-ukraine
["", "blogg"] | <h2 id="trouble-in-the-ex-ussr">Trouble in the ex-USSR</h2>
<p class="publish_date">
20 May 2014
</p>
<p>I am trying to make some sense of the news that come out of Russia and (the) Ukraine. I have been listening a lot to radio: <a href="http://echo.msk.ru/">Echo of Moscow</a> and other news outlets. Now this is my attempt to sum up what i have heard: .</p>
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/Rw0KeJdAZkY" frameborder="0" allowfullscreen=""></iframe>
<p>In some way this meddling is shooting Russia in the foot – if one combines the population and the economy of both Russia and Ukraine then that would make a superpower, whereas taken separately you have two regional powers only. However after the annexation of the Crimea a stronger integration is mostly unlikely to happen within this generation.</p>
<p>How was it possible to push the region into civil war? the population of Easter Ukraine probably feared a stronger integration with the EU – the industrial infrastructure of this region is inherited from the Soviet Union; all the factories are tied to Soviet/Russian industry standards that do not make any sense within the framework of European norms. Germany had a similar problem upon unification and the solution there was to close all East German factories down - the prospect of such a development must have raised a lot of fears.</p>
<h3 id="russian-internal-politics-to-blame">Russian internal politics to blame?</h3>
<p>Mustafa Dzhemilev on the reasons behind Russian intervention in the Ukraine ( see radio interview <a href="http://echo.msk.ru/sounds/1317438.html">here</a> ) ; the Russian establishment is very afraid of a possible revolution, afraid of events similar to those that happened in the Ukraine. An so it goes that Russia decides to push its neighbor into turmoil by supporting the secession of the Russian speaking eastern regions, all that in the hope that the resulting instability will set a detering example for the Russian public, this is what should deter Russians from repeating such an exercise at home. Dmitry Bykov says something quite similar in his poem ( <a href="http://ru-bykov.livejournal.com/1929699.html">here</a> )</p>
<p>At least this makes sense; However that would look like an over-reaction on the part of Russia’s rulers. With a strong economy fueled by high oil prices there would be few causes that could trigger popular unrest. Add to this that many Russians associate political change with social upheaval and anarchy; it seems that Putin’s regime is still within the wider consensus. The Russian state had one strong argument during the pro-democracy protests of 2011/2012 - the message at the time was: don’t rock the boat, now that Russian economy is recovering and Russia has good relations with the west; however now it is the government that is rocking the boat!</p>
<p>There must have been some strong trigger that necessitated strong actions on the part of Russia’s rulers; i guess the Moscow <a href="http://en.wikipedia.org/wiki/Moscow_mayoral_election,_2013">mayoral election of 2013</a> has been a deep shock to the establishment and this is what has put them into panic mode; In these elections opposition parties picked up some 40% of the vote. I guess it showed that at least in Moscow they managed to question the status quo. It is no longer enough to control the media and to rig the election procedure; nowadays social media has a wide reach, grassroots activists can be organized as local election monitors, now this makes vote rigging much more difficult - the concept of a <a href="http://carnegieendowment.org/2005/10/18/essence-of-putin-s-managed-democracy/2a3">managed democracy</a> is only possible when there is a politically passive majority, the terms of the game change when people do care; Of course these fears were amplified by the success of the <a href="http://en.wikipedia.org/wiki/2014_Ukrainian_revolution">Euromaidan</a> revolution.</p>
<p>Or maybe this is just the long anticipated counter reaction to the Russian protests of 2011 ( <a href="http://en.wikipedia.org/wiki/2011%E2%80%9313_Russian_protests">here</a> ) .</p>
<p>In Russian history external threats have been frequently used as a pretexts for the tightening of screws at home. Part of this schema is also implemented right now, for example the current wave of restrictions tries to put some limits on bloggers; that is new - they now take them on as a serious threat.</p>
<p>However restrictions alone don’t make it. Up until now Putin’s rule has had a lot of genuine support, it has been understood as the return to order, an order that came after the stormy nineties; also Russia saw a quite impressive growth in GNI (gross national income) during his rule (up until now). However this support might be waning - it would be put into question if the current rulers are widely perceived as corrupt and as inefficient. This is the card that has been played by a new opposition that has been formed around Navalny. The government is seeking ways to discredit the opposition and to renew its support base. And so they decide to push the Ukraine into political chaos, now they blame Ukrainian liberals for the mess, all with the aim to deter ordinary Russians from a repetition of the same experiments.</p>
<p>Also now every action of the opposition can be explained away as outside interference - these are isolationist tendencies.</p>
<h3 id="alternative-explanations">Alternative explanations</h3>
<p>Alexei Venediktov says that Putin wants to leave his mark onto history, to enter the history books as the gatherer of Russian territories; I doubt that this is a sufficient explanation - the price tag of this project is unrealistically high.</p>
<p>Russian power politics can also be used as a justification: Russia has its black sea fleet stationed in the Crimea (on of the peculiarities of the 1991 breakup of the USSR); everything is fine given that the Ukraine is a non aligned state - once the Ukraine shifts its allegiance to the west this status is really put into question. Russian aspiration towards warm seas is often cited as a major motivation of its foreign policy.</p>
<p>Of course there are problems with this explanation: if the Ukraine is about to default on its debts then neither Europe nor the US will be able to bail out a country with thirty million citizens, I think that the Russian establishment knows that a project of driving out the Russian fleet from Crimea by means of asserting influence on Kiev is totally infeasible.</p>
<p>Since the nineteenth century there has been this discussion between western liberals, who see Russia as being part of the west, and the conservatives who stress that a nominally independent path should be taken. In this context the habitual anti American rhetoric of the Russian establishment is turning into the official ideology; patriotism and Ideological factors become very important as the economy crashes. Maybe the Russian rulers tried to cause a wider crisis in order to assert this ideology, this also seems a bit far-fetched.</p>
<p>Less far-fetched is that the current aggression is exploiting the Russian great power syndrome: Russians remember that the Soviet Union used to be a great power, Putin is seen by many Russians to be the restorer of this lost prestige; so this explains (in part) his lasting popularity in spite of the economic crisis that has come out of this mess.</p>
<h3 id="wider-implications">Wider implications</h3>
<p>another source of the crisis was the prospect of Ukranian default on its debt; the Ukraine had the option of either being bailed out by Russia or by the European union; I don’t think that the European community would be able to bail out a country with thirty million citizens; I guess the same goes for Russia; The fringes of Europe remain a major source of potential crisis - a hundred years after world war I.</p>
<p>Russia also managed to seriously alienate a large number of people in the Ukraine (instead of containing the crisis) - and that was the only country that was really aligned with Russia; this does not serve Russian interests well. (however i doubt that Putin and his friends were aiming at soft power, i think they are not familiar with the concept)</p>
<p>I find it surprising that all these actions that are supposed to strengthen the rule of the current Russian establishment are achieving the opposite result: capital flight will cause an economic downturn (lots of assets are being moved out of Russia) ; with a bad economy people will then raise the question of Putin’s legitimacy. I guess inaction (or rather effortless action, Wu-Wei) were not an option here; as always there is too much at stake for those who are in power …</p>
<p>It is ironic that it is Putin with his aggressive moves who has ruled out any closer ties that might have been formed some day between the former soviet republics;</p>
<h3 id="russian-relations-with-china">Russian relations with China</h3>
<p>Russian conservatives like to see China as a potential ally (it is part of the tendency to stress Russia’s different path from that of the west); however Russian liberals who see themselves as part of the west like to stress the systemic risk of such a move: as China is viewed by liberals as a potential regional opponent.</p>
<p>Russian state television argued that a worsening of relations with the west can be countered with a realignment towards China; I guess they did not ask if it is in the interest of China to be put in a antagonistic relationship with the west; these attempts clearly did not work.</p>
<h3 id="summary">Summary</h3>
<p>Personally i am glad to be watching at all this from far far away;</p>
<hr />
<p>A municipal election can turn into an important event - on condition that a society is standing at a turning point. The East German municipal elections of 1989 were such a moment (big article in German <a href="http://de.wikipedia.org/wiki/Wende_und_friedliche_Revolution_in_der_DDR#Gezielter_B.C3.BCrgerprotest_gegen_Kommunalwahlf.C3.A4lschungen">here</a> )</p>
<p>I guess that this did not escape Mr. Putin, who was a KGB officer stationed in Dresden. This is often described as an unimportant outpost, whoever at the time Dresden was the center of the opposition movement, so Mr. Putin must be very much aware of all this…</p>
Memory leaks revisited
2014-04-27T00:00:00+00:00
http://mosermichael.github.com//all/projects/2014/04/27/preloadut
["", "projects"] | <h2 id="memory-leaks-revisited">Memory leaks revisited</h2>
<p class="publish_date">
27 April 2014
</p>
<h3 id="memory-leaks">Memory leaks</h3>
<p>Leaking memory is a problem that occurs frequently in C/C** programs, Any limited resource will become scarce if it is not properly used; dynamically allocated memory memory must be properly freed up, otherwise a program will run out of it.</p>
<p>A tool like <a href="http://valgrind.org/">valgrind</a> is a very good at solving this problem for smaller programs; if you ask me then this is an important argument in favor of unit testing - checking out unit tests is the stage where valgrind can be used with great efficiency. Actually the <a href="http://en.wikipedia.org/wiki/Unix_philosophy">UNIX Philosophy</a> favors small programs over big monolithic systems - big systems can be composed from little independent pieces; the rational is that it is easier to debug a bunch of small tools rather than that one big monolithic system. Of course this approach is not always doable; however unit testing brings back the advantages of the UNIX Way for larger systems (again for most bugs in most cases).</p>
<p>Debugging large programs is not quite as simple; comprehensive tools like valgrind either slow down the program significantly, or they have a large memory footprint.Some projects start to manage their resources by means of reference counting - the assumption here is that this is easier to trace and validate; still it does not solve the problem.
Other projects like SAMBA have a hierarchy of specialised pooled allocators ( <a href="http://talloc.samba.org/talloc/doc/html/libtalloc__tutorial.html">here</a> ) ; one for the session, one for the request - this approach is similar to reference counting, the problem is not solved, it is moved into a different plane.
Still other managed languages like Java have garbage collection, still you can have a list of object that grows indefinitely without cleanup; the result is the same - the program eventually runs out of memory; so the same problem comes up again, each time it gets a different name.</p>
<p>This article presents some other tools that might help in some situations. Remember that there is no one solution; the requirements of each tool may make its use impractical in some situations.</p>
<h3 id="how-to-download-the-sources--build-the-tools">how to download the sources & build the tools</h3>
<p>Sources can be downloaded as <a href="https://github.com/MoserMichael/cstuff/zipball/master">zip archive</a> or downloaded by the git program.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> git clone https://github.com/MoserMichael/cstuff.git
</code></pre></div></div>
<p>The project can be build on Linux (checked 32 or 64 bit Intel/Amd); it will not work on Cygwin.</p>
<p>Build and install into /usr/local/ directory</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> cd build
./build-dbgmem.sh install
</code></pre></div></div>
<p>Build and install into /usr/alt directory a</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> cd build
` ./build-dbgmem.sh install INSTALL_PREFIX=/usr/alt
</code></pre></div></div>
<h3 id="common-technique-used-here">Common technique used here</h3>
<p>The tools here use a common trick: they install a specialised version of memory allocation functions; the shared library loader is now tricked to call these custom functions instead of the GLIBC memory allocation function
(The substituted functions are: malloc, free, realloc, posix_memalign, valloc; c** also adds new, new[], delete, delete[])</p>
<p>The trick is to instruct the shared library loader to load our specialized shared library first, before loading all other libraries (this is done by setting environment varialble LD_PRELOAD to the full path of our shared library file).
Our special library now implements the memory allocation functions (malloc and free); The trick is that all other shared libraries and the executable will now call our version of malloc and free !
This magic is due to the fact that the Unix shared loader resolves shared library function calls upon first call; when any other shared library, or the executable call malloc, the shared library loader resolves this to the first known implementation.</p>
<p>Now this trick does not work on CYGWIN - CYGWIN is using the Windows shared library loader, and Windows does not support this trick.</p>
<p>An example of this trick is the INITMEM project ( <a href="https://github.com/MoserMichael/cstuff/tree/master/preloadut/initmem">see here</a> )- a very simple tool that sets all newly allocated memory to the value 0xDD and sets all freed memory to the value 0xEE (actually GLIBC does that if you set the environment variable MALLOC_PERTURB_ ). All by itself it is not quite useful, however it shows the concept.</p>
<h3 id="overview-of-the-tools">Overview of the tools</h3>
<h4 id="tracemem---tracing-memory-allocations">TRACEMEM - tracing memory allocations</h4>
<p>One approach is to trace all memory allocations as they happen; a shared library intercepts all call to the memory allocation functions; it directs them to GLIBC memory allocator, however a log entry is written to a file for each call to malloc/free/etc. The log file entry contains pointer value allocated or freed, the size of the memory block, and a few stack frames that describe what part of the program caused the logged even. This is the raw information, when the process exists this information is analyzed, if a memory allocation is not paired with a call to free then this event is reported as a leak.</p>
<p>This trick has a very low memory overhead - nothing is added on top of the memory allocation; however logging each memory allocation takes quite some time, the program will therefore run significantly slower.</p>
<p>GLIBC supports this trick - <a href="http://www.gnu.org/software/libc/manual/html_node/Tracing-malloc.html">see here</a> ; however it is implemented by means of <a href="http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html#Hooks-for-Malloc">malloc hooks</a> ) - a terrible interface that results in critical section lock for every allocation.
TRACEMEM does not add any locks on its own. Another point is that mtrace only records one stack frame - the direct caller of the memory allocation function, whereas TRACEMEM can log several stack frames, so that the context of the memory leak is easier to understand.</p>
<h4 id="dbgmem---memory-debugger-simple-mode">DBGMEM - memory debugger (simple mode)</h4>
<p>this tool again implements its own version of the memory allocation functions - the DBGMEM version of malloc asks asks for more memory than what has been requested by the actual caller of malloc; this added memory is used for a header - this header placed right before the pointer of user memory that is returned by malloc.</p>
<p>This approach has a higher memory overhead if compared with TRACEMEM; however each memory allocation takes less time, as no log record is written to a file. Please note that the current implementation adds a critical section lock; allocations are tracked by means of a double linked list. When the program exits, all remaining entries in this list are classified as leaks.</p>
<p>DBGMEM in simple mode also adds a word of memory before and after the memory block returned to the user; free and delete is therefore able to detect if the memory block has been overwritten.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> linked list of all allocations
----------------------------------------------------------------
| |
V V
[DBGMEM header][guard word][user memory][guard word] [DBGMEM header][guard byte][user memory][guard byte]
[-- memory returned by glibc ----------------------] [-- memory returned by glibc ----------------------]
</code></pre></div></div>
<h4 id="dbgmem---memory-debugger-check-mode">DBGMEM - memory debugger (check mode)</h4>
<p>This tool adds an even larger memory overhead on top of the memory and runtime overhead on top of the GLIBC allocator.
This tool does not place a memory header adjacent to the memory block returned to the user, as this header can be corrupted by a bug in the application. Instead the memory header that maintains its additional information in a Radix tree; the pointer value returned to the application is the key in this Radix tree.</p>
<p>This schema has an additional benefit: for pointers that are located inside an allocated memory block it is relatively easy to look up the information that describes the memory block.
It is possible to add checks for functions like strcpy/memcpy and friends, and check if they overwrite a dynamically allocated memory block;this trick does not always work: for instance the GCC compiler inlines memcpy/strcpy on most occasions.</p>
<h3 id="how-to-use-the-tools">How to use the tools</h3>
<h4 id="using-tracemem">Using TRACEMEM</h4>
<p>There is a helper script - runtracemem; it is used to execute the application that is checked; the script waits until the program exits; on completion of the test run it processes the output log file and generates an html report.</p>
<p>The runtracemem script accepts the following command line;</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[-f reportfile] [-s NUMBER] [-a on|off] [-r <sig_num>] <cmd-to-debug>
Starts a process with TRACEMEM memory allocation tracer. Each memory allocation is traced.
Options:
-f reportfile set name of report file. if not set then the name is $FNAME
-s NUMBER number of stack frames displayed. (default 3)
-a on
-a off if tracing is enabled when process starts (on) or disabled (off); (default on)
-r <sig_num> sets a signal handler; the signal handler toggles tracing mode; (example: if tracing was on then it stops upon signal)
[cmd-to-debug] command line of process that will be started and debugged.
Note: this debugged process will set its core limit to unlimited.
Note: this is a safer way to do what glibc mtrace is doing; also it does not introduce any locks.
</code></pre></div></div>
<p>You can start the process while tracing is disabled, this allows you to skip allocations that are performed once during program start up; tracing can later be turned on once the system has completed its initialisation sequence.
To turn on tracing you can either</p>
<ol>
<li>install a signal handler that will turn on tracing upon reception of signal (option -r <sig_num>)</sig_num></li>
<li>from the application: call mallopt(1004,1) to turn on tracing; call mallopt(1004,0) to turn off tracing.</li>
</ol>
<p>Please note that the generated HTML is best opened with Chrome.</p>
<h4 id="using-dbgmem">Using DBGMEM</h4>
<p>this topic has its own help file: please see <a href="/cstuff/dbgmem/Introduction.htm">here</a></p>
<h4 id="using-dbgmem-simple-allocator-to-debug-cores">Using DBGMEM (simple allocator) to debug cores</h4>
<p>a core file that is produced while using DBGMEM with the simple tool can be analysed for memory corruptions.
This check follows the chain of all alocations and checks if the magic marker at the start and end of the allocation have been overwritten.
In case that a corrupted memory block has been detected, a stack trace that describes where the block has been allocated is shown.</p>
<p>This magic is done by scripting gdb (requires gdb to be compiled with python support). Please be patient, this check takes a lot of time for larger core files.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> gdb <executable file> <core file>
(from gdb promt)
source /etc/dbgmems-read-core.py
</code></pre></div></div>
<p>in one command that would be as follows:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gdb --batch -ex='core-file <path to core file>' -ex='source /etc/dbgmems-read-core.py' <executable program path>
</code></pre></div></div>
The Road to Serfdom
2014-03-31T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2014/03/31/the-road-to-serfdom
["", "ramblings"] | <h2 id="the-road-to-serfdom">The Road to Serfdom</h2>
<p class="publish_date">
31 March 2014
</p>
<p>To what degree should the bureaucracy of the state intervene in economic matters? Every other century, economists come up with new doctrines where this question seems to be the central issue. Consensus seems to swing like a pendulum; Strong state intervention was the method of Mercantilism; later, in the 19th century the pendulum swings back towards liberalism/Laissez-faire economics; during much of the 20th century state intervention was very popular up until the eighties.</p>
<p>“The Road to serfdom” by F. A. Hayek explains a particularly nasty turn of the pendulum, it analyzes the shift towards totalitarian forms of government in the twentieth century; the book is available for <a href="http://www.iea.org.uk/sites/default/files/publications/files/upldbook351pdf.pdf">here</a> . Hayek argues that Germany was leading the way here, and that Britain would be following along the same path (the book was written in 1944)</p>
<p>Much of the book shows how a centrally planned economy leads towards a totalitarian political system. The alternative to economic regulation by means of free market pricing is socialist coercion. The book shows how Germany did slide into Nazi totalitarianism, it is argued that the switch was well prepared.</p>
<p>It follows that: (well I added some details from other sources)</p>
<ul>
<li>A financial crisis, the <a href="http://en.wikipedia.org/wiki/Panic_of_1873">panic of 1873</a> was followed by a <a href="http://en.wikipedia.org/wiki/Long_Depression">long depression</a> ; Germany introduced <a href="http://de.wikipedia.org/wiki/Schutzzollpolitik">protectionist measures</a> in 1878. Hayek argues that these measures were the cause of the formation of monopolies/cartels - <a href="http://de.wikipedia.org/wiki/Gr%C3%BCnderkrach#Weitere_Folgen">here it says</a> that due to a long period of deflation prices were at a low level, and that cartels became instruments of prices fixing, later these cartels would push for anti-competitive measures, industry would become concentrated in the hands of a few firms. In Germany such a state of affairs would later be regarded as the natural result of progress.</li>
</ul>
<p>One remaining question that is not answered is why protectionist measures adopted in France did not lead to the same result, here industry remained more diverse; additional factors like a different structure of industry and finances probably did have some role here.</p>
<p>Another interesting aspect is that the long depression gave rise to a new kind of antisemitism, the novelty was that this was one was based on nationalism and its own racist interpretation of the theory of evolution.</p>
<ul>
<li>The first world war saw the introduction of elements of a centrally planned war economy in Germany. Strict control over raw materials became necessary due to the allied <a href="http://en.wikipedia.org/wiki/Naval_warfare_of_World_War_I">Sea blockade</a> , the Raw materials section/Kriegsrohstoffabteilung of the war ministry was a planning body tasked with control over and distribution of strategic raw materials. German industrialist Walther Rathenau created this planning body; later he also wrote several essays that popularized the idea. The net effect was that a directed/planned economy became to be seen by many as the model remedy for peacetime woes. (Interesting that the whole schema could not have worked without Fritz Haber’s Nitrogen synthesis)</li>
</ul>
<p>This planning body had its peculiarities; companies and raw materials of German owners were not nationalized, the owner of raw materials was restricted in his rights and was obliged to use his property as ordered by the authorities. (however in occupied countries they did loot a lot, later they also started the very nasty policy of abducting foreigners as slaves - to a limited extend as compared to the Nazi regime). The Raw materials sections had the task of control over the global picture: it had to estimate demand of materials vs existing stocks; organize transport capacities; fix prices and organize research of substitute materials; However this body did not assign materials to firms; here industry formed War Economy Corporations/Kriegswirtschafts-Gesellschaften; these non state entities were concentrating materials in one hand and also had the task of assigning specific resources to associated firms; the state had its representatives who would oversee things and who could veto decisions; The state did not centrally plan the production process in all detail, rather it would dictate the ends and means. (<a href="http://www.gutenberg.org/files/21031/21031-h/21031-h.htm">Rathenau / Die Organisation der Rohstoffversorgung</a> ) ; The same principle of government controlled capitalism was later adopted by the Nazi’s.</p>
<p>It is both an irony and tragedy that Walther Rathenau, the author of this scheme, was killed in 1922 by extreme nationalists who, among other things, did not like that fact that he was Jewish.</p>
<ul>
<li>The seeming contradiction between the anarchy of the market and the ‘scientifically organized’ workplace. Technological progress created expectations for social progress, all this called for state intervention or state control. State intervention was seen as a remedy for problems created by the market; The free market did not offer any solutions to problems such as Social security and inner city tenements. Social expenditure rose significantly during the golden years of the Weimar Republic (1924-1929). By 1928 about half of the GDP of Germany was distributed by the state. Hayek argues that such a heavy role of the state in economic matters prepared the ground for an order where the state turned into the sole dictator on every matter. Also a shift in popular attitudes that favored stability over opportunity created a strong preference for jobs in government; Beamtenstaat/civil service state. Liberalism was finished in Germany, long before the Nazi’s took power.</li>
</ul>
<p>The book does not mention several additional facts explained <a href="http://www-wiwi.uni-regensburg.de/images/institute/angegliedert/goemmel/Deutsche_Wirtschaft_und_Wirtschaftspolitik_1914-1945-komplette_Vorlesung.pdf">in these lectures</a> that a high level of public expenses had its reasons, around 8% of the GDP was spent on reparations and payments for debt accrued by the state during the Great War; pensions for invalids and war widows had to be paid; since 1927 they had Social Security guaranteed by law (this measure was enacted by a centrist government, go figure) ;</p>
<p>A big problem of the Weimar Republic was to come up with reparation payments; these had to come from an export surplus, export business became difficult as many European countries introduced protectionist measure after the Great War. Also an export oriented industry requires a large scale of foreign investments (to few savings at home); German central bank set interest rates to 9, against 5 in other countries; (an interesting detail is that the Young Plan obliged Germany to maintain interest rates of at least 5%). High interest rates somehow imply a high level of internal debt.</p>
<p>Before the crisis domestic demand was the engine of growth, during much of the crisis export industry continued to function; this worked until the Banks crashed in June 1931; Preceding events were the crash of the Austrian bank Creditanstalt in 1931 upon press leaks of details on a 100$ million dollar ; a run of investors in 27 June 1931 resulted in bankruptcy of several big German banks; the crashing banks sent the economy into chaos with 32% unemployment by 1932.</p>
<p>Still reparations had to be paid; the Bruehning government could not have devalued the currency, as this would have been a breach of international obligations; deflationary measures would keep prices and wages low (all this with the aim of furthering exports). The result was a deepening of the crisis ; at the same time they got into a habit of overriding parliament and ruling by decree ; the constitution of the Weimar Republic had the famous Article 48 - whereby the President/Reichspraesident could authorize decrees without the approval of parliament, the President was elected by Parliament and not by popular vote.</p>
<ul>
<li>The established Socialist party switched towards representing organized skilled workers (all part of unions), who worked in factories owned by the big trusts. As a result the interests of Labour and industry did converge to a great degree. Now the interest of a few monopoly holders are served at the expense of competition, so some form of socialization would have been the ideal of both bodies; However the interpretation of planning by socialists and monopolists had its differences.</li>
</ul>
<p>It is true that established Socialists became very agreeable, this might explain why the SPD leadership did not start an insurrection when their Prussian government was disbanded by decree on July 20, 1932; despite of them having been in command of the largest police force in Germany.</p>
<ul>
<li>The Nazi party: The traditional unions/Socialist party achieved high salary for skilled workers, the lower middle class was left behind, the Nazi’s filled this electoral niche of a Socialist party for the middle class - now their ideology justified the naked privileges for it’s supporters. The Nazi’s worshiped naked force and propaganda, for they knew the limits of persuasion by argument and reason (these were previously established by the Socialists)</li>
</ul>
<p>The economy of the Nazi state was based on exactly the same schema as that of the war economy during World War I: private ownership was maintained however the state curtails property rights; however Jewish owners were forced to sell their businesses for peanuts - in 1938 this became the official policy (also known as Aryanization)</p>
<p>In industry the tendency was towards weakening of market mechanisms and tightening of state restrictions; In 1933 dividends payments were restricted, profits were supposed to be re-invested. From 1934 the founding of new/extension of existing enterprises must be approved by the state; The state starts to distribute imported raw materials from 1934; Prices and wages were frozen in 1936, independent trade unions were repressed; the Central bank lost its independence in 1937 ;</p>
<p>While being in a state of default, the export oriented economy could not have worked (no means of financing, problem with buying raw materials). So instead the focus was on a buildup towards war, still i don’t understand how this was project was financed. Was it all financed internally?</p>
<p>Still central planning as exercised by the four year plan of 1936-1940 did not cover the whole of the economy; this plan had one specific objective, to prepare the economy for war by furthering autarky - independence from imported raw materials; Therefore it was more direction of investment into strategic materials ; also chemical industry received priority/subsidies for construction of synthetic fuel plants around Buna. There was no central planning office with overriding authority; there was constant conflict between Goehring’s office of the four year plan; Wehrmacht officials who would put prioritize armament production; and Schacht - the then president of the Central bank/Reichsbank.</p>
<p>It remained this way until 1942; that’s when the armament ministry under Speer would be responsible for the entire economy; at this moment they switched to full centralized planning.</p>
<p>So the Nazi system turned into a totalitarian one before they adopted centralized planning. Actually the whole system was exclusively geared towards war; by august 1939 the Nazi’s were bankrupt; foreign currency reserves were totally depleted; the whole construction program was financed by debt alone and the system would have collapsed without the outbreak of WW II.</p>
<p>However with the Soviet Union things were different; collectivization of agriculture and central planning were the point where they switched from an authoritarian system towards a totalitarian one.</p>
<h3 id="my-notes-as-i-read-the-book">My notes as i read the book</h3>
<p>Idea of planned economy became fashionable In Germany after World War I ; in the UK this idea became popular during world war II; both instances the planned war economy was regarded as a positive example. The author says that these ideas lead to the creation of the Nazi system (also Lenin was very much inspired by the German war economy of WW I). “Few are ready to recognize that the rise of Fascism and Nazism was not a reaction against the socialist trends of the preceding period, but a necessary outcome of those tendencies.”</p>
<p>Britain began to move away from a liberal order after WW I. “We have progressively abandoned that freedom in economic affairs without which personal and political freedom has never existed in the past”</p>
<p>Individualism (the basic foundation of capitalism) and collectivism are opposites ; Individualism grew out of the Renaissance “… essential features of that individualism which, from elements provided by Christianity and the philosophy of classical antiquity, was first fully developed during the Renaissance and has since grown and spread into what we know as Western European civilization-the respect for the individual man qua man, that is the recognition of his own views and tastes as supreme in his own sphere, however narrowly that may be circumscribed, and the belief that it is desirable that men should develop their own individual gifts and bents.” ? “Tolerance is, perhaps, the only word which still preserves the full meaning of the principle which during the whole of this period was in the ascendant and which only in recent times has again been in decline, to disappear completely with the rise of the totalitarian state”</p>
<p>“the general direction of social development was one of freeing the individual from the ties which had bound him to the customary or prescribed ways in the pursuit of his ordinary activities … The conscious realization that the spontaneous and uncontrolled efforts of individuals were capable of producing a complex order of economic activities could come only after this development had made some progress” (? but mercantilism was all about control. Adam Smith was a later development)</p>
<p>Individual liberty was prerequisite for growth of science; prior to that “a more extended industrial use of mechanical inventions … were promptly suppressed, and the desire for knowledge was stifled, so long as the dominant views were held to be binding for all”… “The result of this growth surpassed all expectations. Wherever the barriers to the free exercise of human ingenuity were removed man became rapidly able to satisfy ever-widening ranges of desire”
At the turn on the 20th century, progress emboldened men; “With the success grew ambition … and the principles which had made this progress possible in the past came to be regarded more as obstacles to speedier progress”</p>
<p>Principle of liberalism “.. that we should make as much use as possible of the spontaneous forces of society, and resort as little as possible to coercion”</p>
<p>Liberalism - as associated with laissez-faire dogma became discredited; more gradual approach towards monopolies and financial system was needed, but development of these measures took to long for most people; “The attitude of the liberal towards society is like that of the gardener who tends a plant and in order to create the conditions most favorable to its growth must know as much as possible about its structure and the way it functions” ? “It might even be said that the very success of liberalism became the cause of its decline. Because of the success already achieved man became increasingly unwilling to tolerate the evils still with him which now appeared both unbearable and unnecessary”</p>
<p>Success of industrialization became to be taken for granted; interest in and appreciation of how the system worked declined,
“habits of thought of the natural scientist and the engineer, how these at the same time tended to discredit the results of the past study of society which did not conform to their prejudices, and to impose ideals of organization on a sphere to which they are not appropriate.” As a result they did not trust ‘spontaneous forces and individualism’ - the new intellectual trend was ‘collective and conscious decisions/planning’;</p>
<p>From 1870 onward Britain became less of an ‘exporter of ideas’ (like liberalism) instead they ‘imported ideas’ from Germany; for example idea of Socialism became important in Germany before it came to England.</p>
<p>Early Socialist thinkers all had an authoritarian streak, (collectivism vs individualism). Tocqueville: “Democracy extends the sphere of individual freedom, socialism restricts it. Democracy attaches all possible value to each man; socialism makes each man a mere agent, a mere number. Democracy and socialism have nothing in common but one word: equality. But notice the difference: while democracy seeks equality in liberty, socialism seeks equality in restraint and servitude”</p>
<p>During the revolutions of 1848, Socialists and democrats became allies (young Marx); many socialist speak of ‘freedom from necessity’ - but that is not the same as liberty and freedom of expression. Walter Lippmann: “… when men retreat from freedom to a coercive organization of their affairs. Though they promise themselves a more abundant life, they must in practice renounce it; as the organized direction increases, the variety of ends must give way to uniformity. That is the nemesis of the planned society and the authoritarian principle in human affairs.”</p>
<p>Points out similarities between Communism and Fascism; both recruited from among Socialists; for both the real enemy were the Liberals.</p>
<p>His definition of socialism: instance of collectivism that wants to abolish private ownership on means of production + introduction of central planning. Planning defined as “central direction of all economic activity according to a single plan, laying down how the resources of society should be ‘consciously directed’ to serve particular ends in a definite way”; wants to get rid of competition/markets. Ideal is that planning is just and egalitarian, the same means of management can be used to implement an order where the few benefit from the toil of the many.</p>
<p>Liberals want to “create conditions under which the knowledge and initiative of individuals is given the best scope so that they can plan most successfully”; tries to create legal framework so that competition will be beneficial to all (laws like limit to working hours, fraud prevention, prohibition of harmful substances); competition is the “only method by which our activities can be adjusted to each other without coercive or arbitrary intervention of authority” ; market is the only authority to set prices or quantities, price used as the method of coordination between individuals.</p>
<p>Both competition and central direction become poor and inefficient tools if they are incomplete (? maybe each of them has its scope?)</p>
<hr />
<p>arguments for planning: competition is made expedient by technological change; (well another one was tendency towards big conglomerates/monopolies); large firms are supposedly better than small ones because of economies of scale/mass production; complexity of society requires planning for creation of coherent picture.</p>
<p>Also narrow specialist will be under the illusion that their own concern will get higher priority with a planned economy; that makes a lot of people, as there are many narrow specialists.</p>
<p>They argued that “only choice left to us is between control of production by private monopolies and direction by the government.” Argues that the rise of monopolies was not inevitable - it was due to policy decisions; collusive agreements and pressure by big big firms that resulted in decisions that shut out smaller competitors. In Germany since 1878 cartels were encouraged (is he is referring to the protectionist measures of <a href="http://de.wikipedia.org/wiki/Schutzzollpolitik">Shutzzoll</a> - this lead to large cartels fixing prices among themselves; also he forgot to mention the <a href="http://de.wikipedia.org/wiki/Gr%C3%BCnderkrach">Gruenderkrach</a> ?) He says that Britain introduced protectionist measures in 1931 - and that this also lead straight toward the creation of monopolies.</p>
<p>Argues that decentralization becomes important because of complexity of modern society, as central planning of each detail becomes impossible (too much information processing); only competition/market mechanism remain as coordination mechanism of decentralized system.</p>
<hr />
<p>Common feature of collectivist system: want to employ planning in order to concentrate resources on one strategic goal (aka ‘common purpose’) (refuse to “recognize autonomous spheres in which the ends of the individuals are supreme”) ; movements differ in the definition of that strategic goal. However one goal can’t satisfy them all - “The welfare of a people, like the happiness of a man, depends on a great many things that can be provided in an infinite variety of combinations. It cannot be adequately expressed as a single end, but only as a hierarchy of ends, a comprehensive scale of values in which every need of every person is given its place”. So planning presupposed a complete ethical code binding for all, so that planners can determine priority of task; so far ethical codes were limiting in their nature, not prescribing what to do in a total sense. Constraints of information processing limits make it impossible to valuate all multitudes of things objectively according to such an ethical code, even if it exists.</p>
<p>Individualism is based on assumption that there is nobody suited better than the individual to decide about priorities/valuations;</p>
<p>Social action starts where the interest of individuals coincides (it can direct positively only consensus can be achieved by discussion) still the aims of the individual remain the end and measure, individual does not dissolve within the collective.
The state becomes oppressive, when the state forces its hands in matters on the matter here the common aims end.
“Once the communal sector, in which the state controls all the means, exceeds a certain proportion of the whole, the effects of its actions dominate the whole system.” example: Germany in 1928 more than of the GDP (53%)
He argues that with such a state of the economy it was impossible to govern Germany democratically; that was long before the Nazi’s.</p>
<p>Society with central planning will have to decide upon the common objective of the plan; parliament would be unable to reach agreement (parliamentary process can’t be used for administrative tasks), pressure would mount and put technocrats in charge of decisions, taking the place of elected officials) - this would be the end of democracy. Parliament has no problem with complex laws - these are rules that a majority can reach agreement for or against; with economic planning interests are much more divergent and harder to reconcile. Argues that parliament at its best could only replace the chief planner; however it cannot direct this mechanism.</p>
<p>the “point .. is not that dictatorship must inevitably extirpate freedom, but rather that planning leads to dictatorship because dictatorship is the most effective instrument of coercion and the enforcement of ideals, and as such essential if central planning on a large scale is to be possible” ? “it is not the source but the limitation of power which prevents it from being arbitrary “ => democracy without limit of its power becomes arbitrary.</p>
<hr />
<p>Law</p>
<p>“government in all its actions is bound by rules fixed and announced beforehand-rules which make it possible to foresee with fair certainty how the authority will use its coercive powers in given circumstances, and to plan one’s individual affairs on the basis of this knowledge”</p>
<p>“discretion left to the executive organs wielding coercive power should be reduced as much as possible”</p>
<p>If the state tries to limit interference into economic activity, it will work with general laws that do not prescribe everything.
“if the actions of the state are to be predictable, they must be determined by rules fixed independently of the concrete circumstances which can neither be foreseen nor taken into account beforehand: and the particular effects of such actions will be unpredictable. If, on the other hand, the state were to direct the individual’s actions so as to achieve particular ends, its action would have to be decided on the basis of the full circumstances of the moment and would therefore be unpredictable.”</p>
<p>If the state now directs economic activity directly to specific ends, it will also tend towards the idea of arbitrary government;
the government will have to decide between conflicting interests, and decide which request has more merit (should a nurse or a doctor earn more ?); this can only be done by arbitrary decision. Distinction of rank will guide these decision and become part of the ‘law of the land’ ; these choices will create discrimination (one man is allowed what another man is forbidden), this is counter to the idea of formal law.</p>
<p>With planning “the use of the government’s coercive powers will no longer be limited and determined by pre-established rules … If, however, the law is to enable authorities to direct economic life, it must give them powers to make and enforce decisions in circumstances which cannot be foreseen and on principles which cannot be stated in generic form”</p>
<p>Even an ideal form of planning inevitably restricts freedom “If the state is precisely to foresee the incidence of its actions, it means that it can leave those affected no choice.” If the state decides upon everything, then everything becomes a political decision.</p>
<hr />
<p>Liberty</p>
<p>Planners want to maintain democracy - if it does not interfere with economic planning; lack of worry about economic matters is supposed to free the mind to things of higher importance; This argument ignores the fact that money is an instrument for choice; if all choice is done by planners then there will be nothing to choose by the rest of us. “And whoever controls all economic activity controls the means for all our ends, and must therefore decide which are to be satisfied and which not.”</p>
<p>“The so-called economic freedom which the planners promise us means precisely that we are to be relieved of the necessity of solving our own economic problems and that the bitter choices which this often involves are to be made for us.”</p>
<p>The state monopoly would put consumers at the mercy of that monopoly, as there would be no choice left. Also choice of workplace would be severely limited; all determined by planners. An individual would have no way to challenge this assignment; less respect for individual choices.</p>
<hr />
<p>Equality?</p>
<p>choice is between “system where it is the will of a few persons that decides who is to get what, and one where it depends at least partly on the ability and enterprise of the people concerned and partly on unforeseeable circumstances.”
However the author does admit that private property + inheritance does create unequal opportunities ;-) still the poor have more choices than the rich under a dictatorship.</p>
<p>“in transferring all property in the means of production to the state they put the state in a position whereby its action must in effect decide all other incomes”</p>
<p>“It is only because the control of the means of production is divided among many people acting independently that nobody has complete power over us”</p>
<p>“It may be bad to be just a cog in an impersonal machine; but it is infinitely worse if we can no longer leave it, if we are tied to our place and to the superiors who have been chosen for us. Dissatisfaction of everybody with his lot will inevitably grow with the consciousness”</p>
<p>“As soon as the state takes upon itself the task of planning the whole economic life, the problem of the due station of the different individuals and groups must indeed inevitably become the central political problem. … There will be no economic or social questions that would not be political questions in the sense that their solution will depend exclusively on who wields the coercive power, on whose are the views that will prevail on all occasions.”</p>
<p>“socialists everywhere were the first to recognize that the task they had set themselves required the general acceptance of a common Weltanschauung, of a definite set of values. It was in these efforts to produce a mass movement supported by such a single world view, that the socialists first created most of the instruments of indoctrination of which Nazis and Fascists have made such effective use.” ? “The idea of a political party which embraces all activities of the individual from the cradle to the grave, which claims to guide his views on everything, and which delights in making all problems questions of party-Weltanschauung, was first put into practice by the socialists”</p>
<p>Traditional socialist parties represent the skilled workers, were dominated by Marxist dogma that society is divided into captialists and industrial workers; trade unions achieved relatively high salary for its support base. They ignored the envy of those left behind, the army of clerks, administrative workers, small officials, teacher - the new middle class, these became the focus of the Nazi’s. Labour parties converged with/accepted features of the liberal order, aggressive Nazi’s won because their ideology would justify the privileges of its support base. They “knew that the strongest group which rallied enough supporters in favor of a new hierarchical order of society, and which frankly promised privileges to the classes to which it appealed, was likely to obtain the support of all those who were disappointed because they had been promised equality but found that they had merely furthered the interest of a particular class”</p>
<hr />
<p>Social security</p>
<p>“It has been well said that while the last resort of a competitive economy is the bailiff, the ultimate sanction of a planned economy is the hangman”</p>
<p>Social security of socialism comes at the price of loss of freedom/subordination - similar to the army. Security of the producer means security against underbidding by competition; the consumer will have to pay for the security of the producer; security for some is traded against insecurity for others.</p>
<p>Regulation and protectionist measures benefit the few and make them more secure, while making the multitude less secure. In such a setting the certainty of entitlement is chosen over freedom/chance of success. Attitudes change, and so does society. “The younger generation of to-day has grown up in a world in which in school and press the spirit of commercial enterprise has been represented as disreputable and the making of profit as immoral, where to employ a hundred people is represented as exploitation but to command the same number as honourable.”</p>
<p>Example of Germany: organized top down as a Beamtenstaat/civil service state; “not only in the Civil Service proper but in almost all spheres of life income and status were assigned and guaranteed by some authority.” In Germany “where the alternative to security in a dependent position is a most precarious position, in which one is despised alike for success and for failure, only few will resist the temptation of safety at the price of freedom. Once things have gone so far, liberty indeed becomes almost a mockery, since it can be purchased only by the sacrifice of most of the good things of this earth. In this state it is little surprising that more and more people should come to feel that without economic security liberty is “not worth having” and that they are willing to sacrifice their liberty for security.”</p>
<p>Ben Franklin: “Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety.”</p>
<hr />
<p>prevalent morals</p>
<p>“Just as the democratic statesman who sets out to plan economic life will soon be confronted with the alternative of either assuming dictatorial powers or abandoning his plans, so the totalitarian dictator would soon have to choose between disregard of ordinary morals and failure.”</p>
<p>Morals also serve to sanctify prevalent practices in a society; “The ruling moral views will depend partly on the qualities that will lead individuals to success in a collectivist or totalitarian system, and partly on the requirements of the totalitarian machinery.”
Communist/fascists movement comes to power when society is in crisis; people seek a determined and strong leader who gets-things-done (against an inefficient parliamentary system).
Here comes the new model party with its own paramilitary organization around it, one that stresses hirarchy and discipline; one that has no moral problems (where other socialist were reluctant to take power and introduce coercion, the new model army has no such problems).</p>
<p>“… question can no longer be on what a majority of the people agree, but what is the largest single group whose members agree sufficiently to make unified direction of all affairs possible; or, if no such group large enough to enforce its views exists, how it can be created and who will succeed in creating it.”</p>
<p>in recruiting this party you need a large number of like minded men;</p>
<ul>
<li>among more educated people you find a too high differentiation of tastes, the party therefore tends towards a lower common denominator.</li>
<li>for large numbers they must “obtain the support of all the docile and gullible, who have no strong convictions of their own but are prepared to accept a ready-made system of values if it is only drummed into their ears sufficiently loudly and frequently. It will be those whose vague and imperfectly formed ideas are easily swayed and whose passions and emotions are readily aroused who will thus swell the ranks of the totalitarian party … “the desire of the individual to identify himself with a group is very frequently the result of a feeling of inferiority, and that therefore his want will only be satisfied if membership of the group confers some superiority over outsiders “</li>
<li>common feature: need an enemy to drum up support (negative program is more attractive than a positive one). The “Jew had come to be regarded as the representative of capitalism because a traditional dislike of large classes of the population for commercial pursuits had left these more readily accessible to a group that was practically excluded from the more highly esteemed occupations … The fact that German antisemitism and anti-capitalism spring from the same root is of great importance for the understanding of what has happened there”</li>
</ul>
<p>The party defends the narrow interest of a select core, only natural because collectivism is only possible with small number of participants. In parallel a strong preference for narrow national interests evolves; this serves the defense of the inner parties realm.</p>
<p>“by concentrating power so that it can be used in the service of a single plan, it is not merely transferred but infinitely heightened; that by uniting in the hands of some single body power formerly exercised independently by many, an amount of power is created infinitely greater than any that existed before, so much more far-reaching as almost to be different in kind.” … “the ‘substitution of political for economic power’ now so often demanded means necessarily the substitution of power from which there is no escape for a power which is always limited. What is called economic power, while it can be an instrument of coercion, is in the hands of private individuals never exclusive or complete power, never power over the whole life of a person. But centralized as an instrument of political power it creates a degree of dependence scarcely distinguishable from slavery.”</p>
<p>End of morality: the ruthless Party adopts that ‘the ends justify the means’ - “there is literally nothing which the consistent collectivist must not be prepared to do if it serves ‘the good of the whole’, because the ‘good of the whole’ is to him the only criterion of what ought to be done”</p>
<p>Personality of politicians “must himself be prepared actively to break every moral rule he has ever known if this seems necessary to achieve the end set for him. Since it is the supreme leader who alone determines the ends, his instruments must have no moral convictions of their own. They must, above all, be unreservedly committed to the person of the leader; but next to this the most important thing is that they should be completely unprincipled and literally capable of everything.”</p>
<hr />
<p>Propaganda and the end of truth</p>
<p>For efficiency of the system it has to persuade its subject to adopt its ends - by means of propaganda it tries to achieve the Gleichschaltung of all minds. “striving for equality by means of a directed economy can only result in an officially enforced inequality-an authoritarian determination of the status of each individual in the new hierarchical order; that most of the humanitarian elements of our morals, the respect for human life, for the weak and for the individual generally, will disappear”.</p>
<p>Totalitarian propaganda is “destructive of all morals because they undermine one of the foundations of all morals, the sense of and the respect for truth.” - truth can’t just be abolished, it has to be replaced by a new truth, it needs an official doctrine/a system that allows its subjects to rationalizes the ‘wise’ decisions of the leadership.
“Plato’s “noble lies” and Sorel’s “myths” serve the same purpose as the racial doctrine of the Nazis or the theory of the corporative state of Mussolini. They are all necessarily based on particu1ar views about facts which are then elaborated into scientific theories in order to justify a preconceived opinion.”</p>
<p>Totalitarian propaganda assigns ‘new’ meanings to familiar words; ‘freedom’ becomes the freedom of the planners. “Gradually, as this process continues, the whole language becomes despoiled, words become empty shells deprived of any definite meaning, as capable of denoting one thing as its opposite and used solely for the emotional associations which still adhere to them.”</p>
<p>The system will have to suppress dissents and doubt, all means are “used exclusively to spread those views which, whether true or false, will strengthen the belief in the rightness of the decisions taken by the authority; and all information that might cause doubt or hesitation will be withheld. The probable effect on the people’s loyalty to the system becomes the only criterion for deciding whether a particular piece of information is to be published or suppressed”</p>
<p>the system will have to control every aspect of live; it will have to suppress every hint of a doubt, also it often does extend to neutral subjects as physics and mathematics, everything becomes politicized. “There must be no spontaneous, unguided activity, because it might produce results which cannot be foreseen and for which the plan does not provide. It might produce something new, undreamt of in the philosophy of the planner.”</p>
<p>“Once science has to serve, not truth, but the interests of a class, a community, or a state, the sole task of argument and discussion is to vindicate and to spread still further the beliefs by which the whole life of the community is directed. As the Nazi Minister of Justice has explained, the question which every new scientific theory must ask itself is: ‘Do I serve National-Socialism for the greatest benefit of all?’ The word truth itself ceases to have its old meaning”
“The general intellectual climate which this produces, the spirit of complete cynicism as regards truth which it engenders, the loss of the sense of even the meaning of truth… are all things which one must personally experience-no short description can convey their extent.”</p>
<p>“The tragedy of collectivist thought is that while it starts out to make reason supreme, it ends by destroying reason because it misconceives the process on which the growth of reason depends”
Principle of conscious control places some individuals into the position of absolute rulers (a position of hubris), whereas reason/rationality can only grow out of individualism (a position of humility)</p>
<hr />
<p>The Socialist roots of Nazism</p>
<p>Bismark introduced protectionist measures after 1878 (as a consequence of the Gruenderkrach/Crash of 1873) ; Trusts/conglomerates formed as a result (for fixing prices !); Some German socialists (Lensch) see these conglomerates as a higher/socialist forms of organization.</p>
<p>Socialists in Germany of World War I : on one side were for organized/planned economy (in practice they had a planned war economy), on the other hand were supporting nationalistic aims.
Walther Ratenau held important posts in the planning bodies (raw material dictator).
Lensch interpreted WW1 as a conflict between socialist/organized Germany and liberal/individualistic Britain; Lensch - “The state has undergone a process of socialization, and Social Democracy has undergone a process of nationalization. “
Oswald Spengler “Old Prussian spirit and socialist conviction, which to-day hate each other with the hatred of brothers, are one and the same.” also identifies liberalism with Britain as opposed to Prussia; the Prussian idea of a corporatist state (Beamtenstaat) that subordinates the individual.
Moeller van den Bruck “The fight against the capitalistic order, according to this view, is a continuation of the war against the Entente with the weapons of the spirit and of economic organization, the way which leads to practical socialism”</p>
<hr />
<p>And in Britain … ?</p>
<p>By end of the 1930ies Liberalism has gotten into disrepute/perceived as old fashioned by most intellectuals in Britain (outdated/the old thing); very similar mood to what happened during WWI in Germany
this was presented as an inevitable trend ! “The conviction that this trend is inevitable is characteristically based on familiar economic fallacies-the presumed necessity of the general growth of monopolies in consequence of technological developments, the alleged “potential plenty”, and all the other popular catchwords which appear in works of this kind”</p>
<p>Also in Germany they had “scientists agitating for a ‘scientific’ organization of society … Few people remember that in the modern history of Germany the political professors have played a role comparable to that of the political lawyers in France.”
“The influence of these scientist-politicians was of late years not often on the side of liberty: the “intolerance of reason” … and the contempt for anything which was not consciously organized by superior minds according to a scientific blueprint, were phenomena familiar in German public life” .. many scientist (natural sciences) and engineers accepted the Nazi’s with enthusiasm; much more readily than social sciences !</p>
<p>Example of British commies/nazi’s E.H. Carr ; C.H. Waddington ; Laski - Ford in America. The author is alarmed that both organized capital and organized labour are pointing working toward the same end of ‘scientific organization’ and control.
“They do this through their common, and often concerted, support of the monopolistic organization of industry; and it is this tendency which is the great immediate danger.”</p>
<p>Monopolies were pushing towards “corporative society in which the organized industries would appear as semi-independent and self-governing ‘estates’. Monopolistic capitalist got popular support by “letting other groups participate in their gains or, and perhaps even more frequently, by persuading them that the formation of monopolies was in the public interest”
“the public generally nowadays accept the ability to pay higher wages as a legitimate argument in favor of monopoly”</p>
<p>Argues that state control over monopolies give better result than direct state management. This would also limit sphere of influence of monopolies, where these are required, and argues for to “stimulate the invention of substitutes which can be provided competitively”;</p>
<p>Labour movement got coopted by monopolies; a monopoly could offer higher wages (all at the expense of smaller competitors); convergence of interests between big business and Labour.</p>
<hr />
<p>Argues that the current generation demands a rational explanations for every decision but that there are limits of what can be explained. Market pricing is a regulating entity: many decisions are guided by considerations of price (such as choice of occupation); therefore the mechanisms of how prices are exactly set is impossible to understand in exact terms; Same with planning: to explain a particular decision sometimes a great number of things beside the scope of that decision have to be considered, so that it can’t be effectively explained at all.</p>
<p>“It was men’s submission to the impersonal forces of the market that in the past has made possible the growth of a civilization which without this could not have developed; it is by thus submitting that we are every day helping to build something that is greater than anyone of us can fully comprehend”. Speaks about limits of knowledge (in social sciences) - total social control would destroy western civilization.</p>
<p>Labour argued for continuation of WW2 planning in peace time, problem: “individual freedom cannot be reconciled with the supremacy of one single purpose to which the whole society must be entirely and permanently subordinated.” - exception is War.</p>
<p>Example: transition from War economy to peacetime would make many specialized jobs in armament industry redundant. This adjustment can be achieved by monetary means - or by socialist coercion. Changed requirements will create unemployment, that could be delayed, but paying people to work just for the purpose of working would lead to inflation. Instead of increasing the GDP this would amount to redistribution and who would loose out would be very embittered.</p>
<p>“It should never be forgotten that the one decisive factor in the rise of totalitarianism on the Continent, which is yet absent in this country, is the existence of a large recently dispossessed middle class.”</p>
<p>“morals are of necessity a phenomenon of individual conduct, but also that they can exist only in the sphere in which the individual is free to decide for himself and called upon voluntarily to sacrifice personal advantage to the observance of a moral rule ?. Only where we ourselves are responsible for our own interests and are free to sacrifice them, has our decision moral value.”</p>
<p>“Freedom to order our own conduct in the sphere where material circumstances force a choice upon us, and responsibility for the arrangement of our own life according to our own conscience, is the air in which alone moral sense grows and in which moral values are daily re-created in the free decision of the individual. That in this sphere of individual conduct the effect of collectivism has been almost entirely destructive, is both inevitable and undeniable. A movement whose main promise is the relief from responsibility cannot but be anti-moral in its effect”</p>
<p>“It is true that the virtues which are less esteemed and practiced now-independence, self-reliance, and the willingness to bear risks, the readiness to back one’s own conviction against a majority, and the willingness to voluntary co-operations with one’s neighbours - are essentially those on which the working of an individualist society rests. Collectivism has nothing to put in their place, and in so far as it already has destroyed them it has left a void filled by nothing but the demand for obedience and the compulsion of the individual to do what is collectively decided to be good.”</p>
<hr />
<p>Planning and international relations; competition for economic resources on international level would now have to be negotiated among armed state actors; what once was the subject of trade would then be in domain of politics.
some call for planning by super national planning authority; but “There need be little difficulty in planning the economic life of a family, comparatively little in a small community But as the scale increases, the amount of agreement on the order of ends decreases and the necessity to rely on force and compulsion grows.”
Larger groups of people means that it is much harder to reach agreement; also harder to argue for sacrifices that benefit a group of people in a far away country.</p>
<p>Says that international planning would result in naked rule of coercion (Grossraumwirtschaft of the Nazi’s)</p>
<p>Planning on international scale would have to cope with differences in living standards - these must be ordered by some measure of merit these decisions will perceive as having been imposed by a hostile power (by those affected by the decision). The planners/new rulers would then resort to coercion.
“it is impossible to be just or to let people live their own life if the central authority doles out raw materials and allocates markets, if every spontaneous effort has to be “approved” and nothing can be done without the sanction of the central authority.”</p>
<p>Who would be in charge, politician or planner? “Exclusive control of an essential commodity or service … is in effect one of the most far-reaching powers which can be conferred on any authority” No possibility of oversight would be left, everything can be justified by technical necessity. Control over raw materials such as petrol would make every country dependent upon decision of planners; nothing would move without approval of planners (with their own agenda) Predicts that the great powers will use planning bodies as tools of exerting their own agenda upon smaller countries. Planning would force all countries into one centralized state.</p>
<p>How to keep the peace? Hayek argues for political transnational federation (with limited powers) to keep each countries economic interest in check - instead of planning bodies designed to keep political interests in check.
Political organization can seem to remain impartial - because it does not have a direct economical interest.</p>
<p>“Least of all shall we preserve democracy or foster its growth if all the power and most of the important decisions rest with an organization far too big for the common man to surveyor comprehend. Nowhere has democracy ever worked well without a great measure of local self-government, providing a school of political training for the people at large as much as for their future leaders” Local councils are less complex affairs - training grounds for democracy; also smaller countries have much to offer here, so better not push them into the corner. The international order based on law should limit both big and small countries from becoming tyrannical.</p>
<p>“It has always been my conviction that such ambitions were at the root of the weakness of the League of Nations: that in the (unsuccessful) attempt to make it worldwide it had to be made weak, and that a smaller and at the same time more powerful League might have been a better instrument to preserve peace.” Advocates a federation of western countries, with looser associations for more different parts of the world.</p>
<hr />
The old regime and the revolution
2014-02-25T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2014/02/25/tocqueville
["", "ramblings"] | <notextile>The old regime and the revolution</notextile>
<hr />
<p class="publish_date">
25 February 2014
</p>
<p>“The old regime and the revolution” by Alexis De Tocqueville</p>
<p>This book is in the public domain, the English translation is available <a href="https://archive.org/details/oldregimeandrev00tocqgoog">here</a> in electronic form.</p>
<p>An extremely interesting book, Tocqueville makes the impression of having been an extremely good listener and observer, this books explains the events that caused the Revolution, it does so by means of examining the history of the Ancient regime, it is better understood in context with the preceding centuries.</p>
<p>The causes of the French Revolution are explained as follows:</p>
<ul>
<li>Aristocracy was exempt from taxes, and the nobility were not burdened by the task of administering the country, such were the result of <a href="http://en.wikipedia.org/wiki/Cardinal_Richelieu#Chief_minister">Richelieu’s reforms</a> - the aristocracy ceded power to the centralized state and got tax exemptions in return. Privilege without responsibility caused resentment; The middle-classes (wealthy peasants that bought land) all abandoned the land and got to the cities, where they fought over government posts. Society was very fragmented, all this as a result of unequal taxation. French noblemen behaved like a closed caste; by marrying a bride of lower origin any offspring would have lost these precious privileges. Many noblemen lived far away in Paris by means of seigniorial rents - money that they could exert only because of their status; this was very much resented. As commoner grew wealthy, the aristocracy became less rich, so the third estate really was sick of its minders.</li>
<li>every aspect of live in each province was micromanaged directly by the Royal Council in Paris; Even the repair of a local church roof had to be approved by bureaucrats in Paris; therefore even the simplest matter took a long time (not less than a year)</li>
<li>the Royal council had both judiciary/executive/legislative functions; so the result was a mess of contradictory regulations, ad-hock exemptions were very common, state servants had immunity from prosecution, all this bred disrespect for the law.</li>
<li>the revolution was well prepared: Centralization created a class of commoners/administrators (nobility could not be bothered - jobs like this were held in low esteem); an independent and outspoken judiciary did exist, it nevertheless had nothing to say because the Royal Council would overrule them on any matter; Political Philosophers, who did not have any responsibility or experience in matters of government, turned into leaders of discontent; their criticism was quite radical, it was of a general nature and tended to question the basics of society (for whatever reasons books were allowed to question authority of general principles ; however criticism of current events was forbidden, newspapers were strictly censored). The regime was very tolerant when dealing with the educated classes, while it had no scruples with commoners.</li>
<li>The reign of Louis XVI was a liberal period that saw many reforms; the king had to take public opinion into account. Reforms of 1878: regional parliaments were established, the old and new systems were busy fighting each other. A lot of resentment was caused when newly introduced measure were taken back (like with abolition of statute labour for road construction - <a href="http://en.wikipedia.org/wiki/Corv%C3%A9e">Corv?e</a> ; also second term and dismissal of <a href="http://en.wikipedia.org/wiki/Jacques_Necker">Jacques Necker</a> )</li>
<li>“Nations that have endured patiently and almost unconsciously the most overwhelming repressions, often burst into rebellion against the yoke the moment it grows lighter … Evils which are patiently endured when they seem inevitable, become intolerable when once the idea of escape from them has been suggested”</li>
<li>The Catholic Church was associated/entrenched in the old order. Few restraints remained once it has been swept away by the Revolution.</li>
</ul>
<p>However, despite all this, the time that preceded the revolution was one of fast economic growth, the French public was very educated (at least at its center, in Paris), preceding reforms and stability were said to have resulted in a quick growth of industry - all that strengthened the third estate. I wonder what course history would have taken without the disruption of the French revolution, France would have dominated continental Europe and Britain might not have been able to turn into the dominating power of the nineteenth century; also a stronger France might have delayed or even prevented the unification of Germany.</p>
<p>However the book barely mentions the fact that a mountain of debt has been accrued as a result of foreign policy adventures: the seven year war and later involvement in the American Revolution were the causes (during American Revolution the monarchy spent about 1.3 billion livres - in today’s money that’s thirteen billion U.S. Dollars ! <a href="http://en.wikipedia.org/wiki/France_in_the_American_Revolutionary_War#Financial_aspects">here</a> ). The house of Habsburg went bankrupt as a result of the thirty year war; the house of Bourbon got broke as a result of its competition with Britain. In 1781 - by an unprecedented act of transparency the Crown published <a href="http://en.wikipedia.org/wiki/Compte_rendu">its expenses</a> , now the books were cooked in order to conceal the true state of affairs. It gradually became impossible get new credit once the true scope of the problem surfaced later in the decade. The book also does not mention the failed harvests of 1783 and 1788 <a href="http://alphahistory.com/frenchrevolution/harvest-failures/">here</a> ; probably as such this was not something out of the ordinary, however it occurred in combination with the bad fiscal crisis. As usual latent tensions mount once the economy suddenly tanks.</p>
<p>Also the author describes the continuity of French governance - after 1800 they had the same highly centralized state that was commanding everything down from Paris;</p>
<p>However the big achievement of the Revolution was equality of all citizens before the law, Egalite was the lasting result. Another lasting result was a popular appreciation of freedom and liberty.</p>
<p>The other big change was the invention of the nation state, a King of old did not have problem with legitimacy; he did not need the auxiliary concept of nationality as a means of legitimizing his rule (however the book does not mention this)</p>
<p>The book also has very interesting observation is about the nature of liberty, and the totalitarian nature of Socialism, he could see as much from the writings of <a href="http://en.wikipedia.org/wiki/%C3%89tienne-Gabriel_Morelly">Morelly</a> written as early as 1755 ! Nothing new under the sun.</p>
<hr />
<h3 id="my-notes-as-i-read-the-book">My notes as i read the book</h3>
<p>Revolution: part I - destroys every remnants of the past, part II - regains portion of the past.
“Non aristocratic societies without liberties may prosper … but will not produce great citizens.”
“Despots acknowledge that liberty is an excellent thing; but they want it all for themselves, and maintain that the rest of the world is unworthy of it.”</p>
<hr />
<p>At first the rulers of Europe thought the French Revolution to be a passing craze; later came the recognition that this is something out of the ordinary.</p>
<p>Anti-clericalism as a lasting trait of the Revolution; philosophers of the 18th century were hostile to the church as an institution; this was intensified by the worldly power/wealth of the church. He says the church came out stronger after being stripped of its worldly belongings.</p>
<p>Revolution did not merely aim at a change in an old government; it designed to abolish the old form of society - any old form of authority/institutions/tradition. “But also created .. an immense central power, which gathered together … particles of authority and influence formerly scattered among a host of secondary powers, orders, classes, professions, families.”</p>
<p>The modern state as the envy of princes … hired officials instead of aristocracy; uniform set of laws instead of local franchises; a single strong government instead of a system of diversified authorities … new order was fragile but this was because of its strength. Mirabeau wrote to the Louis-XVI (!) “.. Richelieu would have liked the idea of forming but one class o citizens; so level a surface assists the exercise of power.”</p>
<p>The ideas and cause of the French revolution united people across national boundaries;
Similar to 30 year war; Schiller observed: one striking effect of Reformation was that it lead to a sudden alliance and warm friendship among nations which hardly knew each other; questions of territory gave way to questions of principle.</p>
<p>There are times when men differ so widely that the bare idea of a common law for all appears unintelligible. The great wonder of the Revolution … is that mankind has reached a point at which … these ideas were readily admitted.</p>
<p>“I have had occasion to study the political institutions which flourished in England, France and Germany during the Middle ages… I have been filled with amazement at the wonderful similarity of the laws established by races so far apart. Each of the three nations enabled me to understand the other two. In all three the government was carried on in accordance with the same principles;: the political assemblies were constituted from the same materials and armed with the same powers; society was divided into the same classes, on the same sliding scale. “</p>
<p>(by the 14th century order converged, by the 18th century the system was in decline - everywhere). Richelieu enforced strict centralization, so the old forms of local franchises/guilds became an inefficient shell. New modes of production were in opposition to the ossified rules. (Not so England - this was a modern state by the 17th century)</p>
<p>“I have seen (Land) registers of the fourteenth and thirteenth centuries, which are masterpieces of method, perspicuity and intellect; the modern ones (19th century) grow more obscure, incomplete and confused. It would seem as though the civilization of society had involved the relapse of the political system into barbarism. All surviving medieval sources of authority had suffered from the same disease; all where decaying and languishing.”</p>
<p>Results of the Revolution: abolished the Feudal system .. “in order to substitute therefore a social and political organization marked by more uniformity and more simplicity, and resting on the basis of the equality of all ranks” (before the law)
Profound change: the old system was justified by Religion and created a host of ideas and custom that justified it.</p>
<p>Why France? In fact the ancient regime was liberal - no serfdom ! land was divided and bought freely by peasants : compare that to Germany where by end of 18th century Serfdom still existed (<a href="http://de.wikipedia.org/wiki/Leibeigenschaft#Leibeigenschaft_in_den_Deutschen_Gebieten">in most places</a> except for Rhine lands)
In France the lands were subdivided into very small parcels, peasants still had to render feudal services, but they were free.
Local government was the domain of officials appointed by the intendant (governor of a province); Nobleman had immunity and privileges but no power - in some limited cases the right of jurisdiction, could levy tolls on fairs and markets (rarely claimed); privilege of hunting; monopoly on mills and wine presses; a tax on land sales/purchases; levied ground rents.
All this was less burdensome for the peasants than in the rest of Europe; however peasants in France were owning land of their own, so these tithes were universally resented; also it was annoying that noblemen got all this for free, as he did not have any authority / not burdened by any obligations !</p>
<p>“… but (the feudal system’s) curtailment was the source of its unpopularity; … the destruction of part of the system rendered the remainder a hundred-fold more odious then the whole”</p>
<hr />
<p>Centralization was a feature of the Ancient Regime:</p>
<p><a href="http://en.wikipedia.org/wiki/Conseil_du_Roi">Royal council</a> - had judicial powers (can overturn court decisions); legislative powers (near to the King); and administrative powers. Its statute is that of advisory body, it was not made up of noblemen. Comptroller general managed home affairs, intendantes represented the state in the provinces. Real power was held by bureaucrats of ‘lower extraction’.
However social live was focused on the Aristocracy - they were the stars of the show; and administrative positions were held in low esteem by Noblemen.</p>
<p>Taxation: were levied by subcontractors; the Royal Council determined amount of taxes and norms for each province.</p>
<p>Conscription: Royal council determined number of conscripts per draft; lottery determined who would serve for six years.
Road construction: again centralized, the Department of Bridges and Roads directed and supervised public works. Policing/Charity/Jurisdiction - all in the hands of the Royal Council; also exerted administrative control over the economy; Council prohibited cultivation of crops in lands that it deemed unsuitable; ordered manufacture of this or that article.</p>
<p>Municipal rights:
Cities were governed in two chambers:
* general assembly was elected (once by popular vote, by 18th century only by notables or guild representatives)
* city corporation: executive elected by general assembly
however people took no interest in these bodies - they had nothing to say; the intendant was the real power.
Cities did not have taxes of their own, did not supervise public works or do contracts.
Cities did have elected municipal governments/magistrates until 1692 (abolished by Louis XVI) ! After the city offices were sold to citizens. Bad for cities: Judges were no longer impartial/independent.
Cities sometimes bought back the right to elect magistrates, but that always taken away from them.</p>
<p>Rural parishes: parish meetings retained more democratic semblance: everyone too part, but they still had nothing to say; intendant had to approve meetings ! Each repair of a church roof would have to be approved via the sub-delegate, back through the Royal Council.</p>
<p>Judiciary:
Courts in France were very independent so they had to be minded; Most Royal edicts stipulated that resulting conflicts would have to be referred to the intendant/council.
In most matters (like taxes, administration) the courts did not have a say, one could not sue the state, and official had immunity ! for all other matters the Council could revoke court decisions.
these interventions fermented discontent.</p>
<hr />
<p>Post revolutionary system of government became very centralized after 1800; meet the new boss, same as the old boss.
State incrementally took up all powers and concentrated it at the center.
Conflicts of <a href="http://de.wikipedia.org/wiki/Parlement">Parlements</a> and crown were around political issues, not administrative questions, this created a vacuum that was grabbed by the government.</p>
<p>“Day after day, the central government conquers new fields of action into which these bodies cannot follow it. Novelties arise, pregnant with cases for which no precedents can be found in parliamentary routine.. Novelties arise, Society, in a fever of activity creates new demands, which the government alone can satisfy, and each of which swells its authority”</p>
<p>Centralization was a preliminary and precursor of the revolution;</p>
<p>The government was increasingly governing every detail, every movement that was made in the country; enormous amounts of paper turned, as a result every moves takes a year.</p>
<p>“Official writers, then as now, affected a colorless, smooth, vague, diffuse style; each writer merged his identity in the general mediocrity of the body to which he belonged; Read a perfect, and you have read an intendant”. “Government officials, none of whom were of noble descent, already formed a class apart, with feelings, traditions, virtues and notions of honor and dignity all their own. They constituted the aristocracy of the new society, ready to take their rank as soon as the Revolution has cleared the way”. “When a nation abolishes aristocracy, centralization follows as a matter of course … Every thing tends towards unity of power, and it requires no small contrivance to maintain division of authority”</p>
<p>the regime was suspicious of independent actors/meddling in public affairs; “It was disturbed by formation of any free society. It could brook no association but such as it had arbitrarily formed … Even manufacturing companies displeased it.
in a word it objected to people looking after their own concerns, and preferred general inertia to rivalry. Still the French could not exist without some form of liberty, they were permitted to discuss as freely as they choose all sorts of general and abstract theories on religion, philosophy, morals and even politics. Provided its agents were not meddled with, the government had no objections to attacks on the fundamental principles of society … Careless about books, it was very strict with regards to journals, and being unable to suppress them, it undertook to make them a government monopoly”</p>
<p>Micromanagement created a flood of regulations, nobody could have been aware of all instructions, so in practice everything was enacted as an exceptions/exemptions; “… the whole principle of the old regime. Strict rules, loosely enforced - such was its characteristics”. This practice lead to wide disrespect for the law,</p>
<p>The state was all powerful, everyone knew that. Prior to the Revolution all sorts of reforms were proposed. “All the schemers wanted to use the central power for the destruction of the existing system, and the substitution of their new plan in its stead: that power alone seemed to them capable of accomplishing so great a task. They all assumed that the power of the state ought to be to be unlimited, and that the only thing needed was to persuade it to use them aright”
.. Nobody expected to succeed in any enterprise unless the state helped him”</p>
<p>“Let no one again express surprise at the wonderful ease with which centralization was re-established in France at the beginning of this century; It has been overthrown by the men of 1789; but its foundations were deep in the mind of its very destroyers, and upon these it was rebuild anew stronger than ever.”</p>
<hr />
<p>Excessive centralization lead to a weakening of the provinces; everyone of note was in Paris, all publishing was done in Paris. “Paris swallowed up the provinces”;
All manufacturing was concentrated around in Paris; this was encourage by lower taxes and it was easier to establish trade-companies in Paris.</p>
<hr />
<p>Absolutism created a host of regulations that were applicable to the whole of France; this was a great unifying force when considering the many many subdivisions of medieval society. The idea of a Universal law for all - absolutism (central rule) prepared the basis for this idea.</p>
<p>Nobility became poorer with every generation; they sold their land and lived from various seigniorial rents. Some commoners became richer - a leveling of society took place, all grew more similar in education and ideas; therefore the privileges of nobility became untenable.</p>
<hr />
<p>the author observes that England had no castes - unlike the rest of Europe (???) so that a nobleman could marry a woman of ordinary descent - not so in France.
Also the English word ‘Gentleman’ became more inclusive with time. A commoner in France could buy himself into Aristocracy; however this did not decrease social barriers at all!</p>
<p>In France of the 14th century - the third estate did matter; it was well represented <a href="http://en.wikipedia.org/wiki/Estates_General_(France)">estates-general</a> , government and defense. This seized when they stopped the estates-gerneral; The <a href="http://en.wikipedia.org/wiki/Taille">Taille</a> tax - aristocrats were exempt from it; this became a big nuisance, once taxes grew due to stronger centralization; Also social status of aristocrats was very high. All these privileges created invisible caste barriers and hatred. Distinctions between commoners and nobility were marked by the fact that one could become a nobleman ;</p>
<p>Exemption of Nobility from taxes: this feature became very valuable, once taxation started to be a significant burden. In Germany the nobility was not exempt from taxes - they paid less than commoners. “Now of all methods that have been devised for the division of nations into classes, unequal taxes are the most pernicious and effective. They tend to isolate each class irremediably; for when the tax is unequal, the line is drawn afresh every year between the taxable and the exempts; the distinction is never allowed to fade. Every member of the privileged class feels a pressing and immediate interest in keeping it up, and maintaining his isolation from the taxable community.
All, or nearly all public measures begin or end with a tax. Hence, when two classes of citizen do not feel the taxes alike, they cease to have common interests and feelings in common; they do not require to meet for consultation; they have no opportunity and no desire to act in concert”</p>
<p>Rich commoners who lived in the city could evade the Taille tax - by leasing their estates to others. this resulted into a flight of the richer peasants from the land, and France had a lot of small towns
The new townsmen desired to gain positions in the bureaucracy - and the bureaucracy did grow tremendously around interest groups (also one had to buy official positions !) Society was divided into an very large number of interest groups.</p>
<hr />
<p>History of taxation:</p>
<p>14th century: Monarch lived of the produce of their domains; taxes were raised only in extraordinary times - everybody had to pay them - and new taxes had to be approved by a vote by Estate-General ! Richelieu’s New deal was that the Aristocracy lost political power to the crown, but gained exemption from taxes. Economically this was not very efficient, as the rich ones got away without paying anything!</p>
<p>”..shows to what straits and dishonest shifts the want of money will reduce a government, however mild it may be, as long as it is unchecked, and fears neither publicity on the one hand, nor revolution - the safeguard of popular liberty - on the other. This history teems with instances of royal property sole, then resumed as inalienable; of violated contracts; vested rights trampled ; public creditors sacrificed at every crisis; the public faith constantly broken”. A frequent trick was to cancel previously granted status of nobility - so the commoners had to pay for the same privileges again !</p>
<p>Trade corporations: one had to be part of a guild and buy the license to trade in his profession from all-powerful state.
Also they were selling public offices for money; and a vast bureaucracy was exclusively busy with selling them.</p>
<p>All this could happen because the state was not accountable to the public, and there was no public discussions / lack of feedback.</p>
<p>“it was in fear lest the nation, whose money the King wanted, should insist upon the restoration of its liberties, that class divisions were kept up; for by this means organized resistance or a common understanding was rendered impossible, and the government was certain of having to deal with each small clique separately”</p>
<hr />
<p>Remaining liberties:</p>
<p>“It is striking to see, in the correspondence of ministers and intendants of the eighteenth century, how quickly this government which was so overbearing and despotic when all was submission, lost its presence of mind at the first show of resistance, was alarmed by the mildest criticism, and terrified at the least noise. On these occasions it stopped short, hesitated, tried to compromise matters, and often withdrew from the contest at the sacrifice of a portion of its legitimate authority”</p>
<ul>
<li>
<p>the nature of the privileges bred dissent</p>
</li>
<li>
<p>on the eve of the Revolution the clergy was for liberty !</p>
</li>
<li>
<p>the middle classes were busy chasing posts and bound by corporate interest, they could not be bothered by liberty</p>
</li>
<li>
<p>judiciary was never servile; a judge had no problem to speak out freely within their stripped jurisdiction (however decisions were overturned by the Royal Council)
“It was from the courts that we learned the only portion of the education of a free people … the principle that all decisions should be preceded by discussion, and subject to appear; the use of publicity, the love of the forums … The king felt bound to assign reason to his edicts; the Councils orders were preceded by long preambles… All the old administrative bodies such as the Treasury board, and the select-man, transacted business publicly, and heard rival petitioners and applicants by counsel. All their habits and forms were so many barriers against the arbitrary power of the sovereign. But the people proper, especially in the rural district, had no means of resisting oppression except by violence”</p>
</li>
<li>… “there was much more liberty than there is now, but it was an irregular and intermittent kind of liberty, bound up with the class system and notions of privileges and exemptions - a sort of liberty which encouraged rebellion against the law as well as against oppression … “</li>
<li>“but if the disorderly and unwholesome sort of liberty prepared the French to overthrow despotism, it unfitted them … perhaps, for replacing it by the peaceful and free government of law”</li>
</ul>
<hr />
<p>Peasants</p>
<ul>
<li>
<p>Aristocracy and land owning commoners (middle class) - they all fled the cities; peasants were left in ignorance</p>
</li>
<li>
<p>as a result of the ignorance of peasants, agriculture was held back; did not progress.</p>
</li>
<li>
<p>Taille tax: each year the amount of money were assigned per parish; one peasant was assigned as the collector - he had to set the norm per household, and was held accountable for the results.</p>
</li>
<li>
<p>peasants ‘assumed the garb of poverty’ in order to escape taxation, so the collector had to spy out and denounce his neighbor.</p>
</li>
<li>
<p>some provinces reserved to tax themselves, here the tax was fixed years ahead and the amount registered - sum determined according to size of estate.</p>
</li>
<li>
<p>Peasants were kept poor and needy on purpose; either as a way to keep them docile, later rulers thought that only by need would peasants be spurred to work.</p>
</li>
<li>
<p>Militia service:</p>
</li>
<li>Repair of roads: peasants had to pay and were forced to do the repairs repairs (Corvee - forced labor tax)</li>
<li>“the government of the old regime, which was so mild and so timid, so fond of formalities and delays in dealing with the upper classes, was often rough and always prompt in dealing with the lower, especially with the peasantry”
burghers were judged by independent tribunal, peasants just arrested by of an intendant.”</li>
</ul>
<p>Summary</p>
<ul>
<li>
<p>tax regulations created gap between nobility and rest of the country; also landed commoners were detached from peasants.</p>
</li>
<li>
<p>society was divided into a huge number of interest groups with narrow interest, with no cohesion among the groups.</p>
</li>
<li>“There was no organization that could resist the government, but there was none that could assist it. So it was that, the moment the groundwork moved, the whole edifice of the French monarchy gave way and fell with a crash”</li>
<li>the people after the revolution failed “.. to succeed in eradicating the false notions, the vicious habits, the bad propensities these masters had either imparted or allowed them to acquire.” so that self government was not effective:</li>
</ul>
<hr />
<ul>
<li>
<p>men of letters/political writers and philosophers did not hold office; “.. all concurred that in one central point … they all started with the principle that it was necessary to substitute simple and elementary rules, based on reason and natural law, for the complicated and traditional customs which regulated society in their time”
… “they had constantly in view a host of absurd and ridiculous privileges, whose burden increased daily, while their origin was growing more and more indistinct; hence they were driven towards notions of natural equality.”</p>
</li>
<li>
<p>due to lack to experience they did not have clear understanding of the problems of government, so they were busy with abstract/general theories of government; therefore there was no inhibition against notion of razing the old order to the ground.</p>
</li>
<li>
<p>General public did not take part in public affairs, so these general theories would not be questioned and became popular. Abstract theorizing was not censored, so ideas became very popular (everybody who had grievances with the existing order were eager listeners)</p>
</li>
<li>
<p>Political writers became the leaders of the Revolution.</p>
</li>
<li>
<p>interesting that rulers never anticipated a revolution - that was not in their domain of experience !</p>
</li>
<li>American Revolution turned into a catalysts of events in France</li>
<li>the author says that a tendency towards grand schemes not based upon experience has remained a trait of French public discourse - long after the end of the Revolution.</li>
</ul>
<hr />
<p>The church/attitudes toward Religion</p>
<ul>
<li>
<p>claims that the Catholic church in France was more tolerant, open, less corrupt and not very oppressive than their foreign peers (?) however it came into conflict with the political writers.
the Church “… recognized a higher authority than individual reason; they allowed of no appeal from reason. It clung to the notion of hierarchy; they insisted on leveling all ranks. The two could never come to an understanding, unless both admitted that political and religious societies, being essentially different, can not be governed by like principles … it seemed to the reformers of the time absolutely necessary to destroy the religious institutions of the time in order to reach the civil institutions”</p>
</li>
<li>
<p>church became a target because …” it had become a political body in defiance of its vocation; it shielded vice in high places, while it censured it among the people; it threw its sacred mantle over existing institutions, and seemed to demand for them the immortality it expected for itself”</p>
</li>
<li>
<p>In America…“Respect for Religion … the best safeguard for political stability and private security”</p>
</li>
<li>characteristic: “they were imbued with one admirable faith which we lack: they believed in themselves…they had a proud reliance in their own selves themselves, and strength;and though this often leads to errors, a people without it is not fit for freedom… these sentiments and passions had become a sort of new religion, which, like many religions…stifled selfishness, stimulated heroism and disinterestedness, and rendered men insensitive to many petty considerations which have weight on us”</li>
<li>“When the French Revolution overthrew civil and religious laws together, the human mind lost its balance. Men knew not where to stop and what measure to observe. There arose a new order of revolutionists, whose boldness was madness, who shrank from no novelty, knew no scruples, listened to no arguments or objections”</li>
</ul>
<hr />
<ul>
<li>
<p>mid 18th century: political writers that do deal with administrative questions - economists (better known as <a href="http://en.wikipedia.org/wiki/Physiocrats">Physiocrats</a> )</p>
</li>
<li>
<p>were not fond/did not aspire towards political liberties; they thought that reforms would have to be lead by the Monarchy (reforms from above)
they were fans of the centralized state + Royal bureaucracy and planned to convert it “the state must govern according to the laws of natural order, says Mercier de la Riviere: ‘on these conditions it should be absolute’”</p>
</li>
<li>
<p>“‘The state’ says Bodeau ‘molds men into whatever shape it pleases’” - interesting idea of totalitarian state; they saw Chinese society as a model for Europe</p>
</li>
<li>
<p>“It is generally believed that the destructive theories known as Socialism are of modern origin. This is an error. These theories are coeval with the earliest economists. … Read the Code de la Nature by Morelly ; you will find there, together with the economic doctrines regarding the omnipotence and boundless rights of the state … community of property, rights of labor, absolute equality, universal uniformity, mechanical regularity of individual movements, tyrannical regulations on all subjects, and total absorption of the individual in the body politic. ‘Nothing,’ says the first article of this code, ‘belongs wholly to any one. Property is detestable, and any one who attempts to re-establish it shall be imprisoned for live, as a dangerous madman and enemy of humanity”.</p>
</li>
<li>
<p>“the only safeguard against despotism which they proposed was public education; for as Quesnay said ‘Despotism is impossible in an enlightened nation’”</p>
</li>
<li>around 1750 the public did not care for political liberties; by 1770 things were different: ‘the provinces began to desire once more to administer their own government. Mans minds became imbued with the notion that the people at large were entitled to a share in their own government”; that’s the moment where revolution became inevitable.</li>
<li>
<p>the idea of equality without liberty appeared later, so that “Hence we resemble the economists of the 1750 more closer than our fathers of 1789”</p>
</li>
<li>what is the origin of the desire for liberty? what are the sources that sustain liberty?
* “when a people is badly governed it desires self-government … is never durable; it passes away with the accident which gave it birth”
* sight of material advantage is not enough; “there are times when it disturbs these blessings for a while, and there are times when their immediate enjoyment can only be secured by despotism”
* “It is the intrinsic attraction of freedom, its own peculiar charm - quite independently of its accidental benefits - which have seized so strong a hold on the great champions of liberty throughout history ; they loved it because they loved the pleasure of being able to speak, to act, to breathe unrestrained, under the sole government of God and the laws. He who seeks freedom for any thing but freedom’s self is made to be a slave.”</li>
</ul>
<hr />
<p>Economics</p>
<ul>
<li>
<p>first half of 18th century was static/lethargic, nothing changed; “no private enterprise of any magnitude was undertaken”; 1750-1789 were very dynamic years !</p>
</li>
<li>
<p>Officials of the state changed: “in 1780 their heads were full of schemes for enriching the public, Road, canals, manufactures, commerce and agriculture above all, absorbed their attention”</p>
</li>
<li>
<p>“a general concern for the ills of the poor, a new sentiment. The state rarely employed violence with the poor, but often remitted their taxes or granted them alms”</p>
</li>
<li>
<p>“public prosperity began to develop with unexampled strides … individuals grew more industrious, more incentive, richer than ever”</p>
</li>
<li>
<p>“of late years maritime trade has made more progress in France than in England; the whole trade of France has doubled in the last twenty years”</p>
</li>
<li>
<p>“Considering the vices of the government and the burdens which weighted upon industry, the spectacle of this great and increasing prosperity is astonishing”
Reasons for this “a government, strong without being despotic, which maintained order everywhere; the other a nation whose upper class were the most enlightened and the freest people on the continent, and in which individuals were at liberty to make money if they could, and keep it when made”</p>
</li>
<li>
<p>Louis XVI had to take public opinion into account, and was quite liberal, if compared to his predecessors !</p>
</li>
<li>
<p>“Measurably with the increase of prosperity in France, mans minds grew more restless and uneasy”; In neighborhood of Paris the taille tax was moderate and well ordered; and Corvee disappear long before 1789,</p>
</li>
<li>
<p>Revolution was strongly opposed in provinces where the old order was backward and suppressive !</p>
</li>
<li>“Nations that have endured patiently and almost unconsciously the most overwhelming repressions, often burst into rebellion against the yoke the moment it grows lighter … Evils which are patiently endured when they seem inevitable, become intolerable when once the idea of escape from them has been suggested”</li>
<li>expenses due to huge public works rose, and government took up large debts; and it would not always pay the interest on the loans ! this drove capitalists and merchants into opposition and strengthened their desire for reforms</li>
</ul>
<hr />
<ul>
<li>
<p>The King had sympathy with the poor, tried to remedy inequalities, and was ready to acknowledge inequalities and problems! This raised discontent to new levels, half measures (like abolition and later re institution of corvee service) had even worse effect;</p>
</li>
<li>
<p>“all this was addressed to the educated classes, in order to prove the merits of measures which certain private interests opposed. As for the people, it was taken for granted that they heard all, but understood nothing”</p>
</li>
<li>“it must be acknowledged that .. benevolence concealed a large share of contempt for them (the poor)”</li>
<li>16th century: nobility took sides with reformation because of selfish interests, the poor embraced it from conviction; French Revolution was different “it was disinterested principle and generous sympathy which roused the upper classes to revolution, while the people where agitated by bitter feeling of their grievances, and a rage to change their conditions. The enthusiasm of the former fanned the flame of popular wrath …”</li>
</ul>
<hr />
<ul>
<li>
<p>Some of the reforms of Louis XVI changed … ‘old and respected customs … they paved the way for the Revolution less by striking down obstacles which stood in its way than by showing the people how it might be brought about”</p>
</li>
<li>
<p>”.. the government took pains to teach the people … that private property was to be regarded with contempt. “ huge amount of public works resulted in many requisitions of smaller tenants “properties thus injured or destroyed were always arbitrarily or tardily paid for, sometimes they were not paid for at all”</p>
</li>
<li>“here were a large number of persons whose own experience taught them that private rights were not for a moment to be balanced against the public interest “.</li>
<li>Price setting (Maximum) and Compulsory sale during bad harvest; arbitrariness when dealing with unrest (administrative tribunal with no right of appeal) - these were the measures of the old regime, equal measures were than used by the Revolution - the King as the teacher of revolutionaries.</li>
</ul>
<hr />
<ul>
<li>
<p>destruction of trade companies and their later partial restoration: disturbed industrial relations.</p>
</li>
<li>
<p>judicial reform of 1788: a large portion of the middle class was involved with bloated courts; this affected the livelihood of many.</p>
</li>
<li>
<p>provincial assemblies of 1787 - became the real governors of the country (superseded the intendant !) -: conflict of institutions, this results in administrative chaos and paralysis
“… how large an influence habit exercised over the working of political institutions, and how much more easily it is to manage their affairs with obscure and complicated laws to which they are used than with a far simpler system which is new to them”</p>
</li>
<li>
<p>Indendants and sub-delegates were stripped of their power, but the office remained - they were supposed to help/supervise the assemblies. instead this lead to entrenched war between institutions and administrative paralysis.</p>
</li>
<li>the idea of separating administration/legislative and executive branch did not occur - this is said to be an idea of the 19th century (?)</li>
<li>Administrative reform of rural districts: the worst problem was unequal taxation; this problem should have been solved before the reforms, as a result existing village authorities were stripped of their powers</li>
</ul>
Wholesale encryption with RSA
2014-02-24T00:00:00+00:00
http://mosermichael.github.com//all/projects/2014/02/24/openssl
["", "projects"] | <h2 id="wholesale-encryption-with-rsa">Wholesale encryption with RSA</h2>
<p class="publish_date">
24 February 2014
</p>
<h3 id="very-short-introduction-to-cryptography">Very short introduction to cryptography</h3>
<p>The following schema is used when large data set are encrypted/decrypted.</p>
<ol>
<li>An initial message exchange is performed using public/private encryption <a href="http://en.wikipedia.org/wiki/Public-key_cryptography">key pairs</a> , note that public and private keys are different. The sending party uses the public key for encryption, while the receiving party is using the private key for decryption; this exchange is very slow. This exchange is secure if no third party is able to guess the private key - the encryption algorithm (such as <a href="http://en.wikipedia.org/wiki/RSA_(algorithm)">RSA</a> ) ensures this much, trust the maths.</li>
</ol>
<!-- -->
<ol>
<li>The sending party generates a symmetric key for encryption and decryption of subsequent data; decryption and encryption of this data is relatively fast, but here the same key is used to both encrypt and to decrypt the data. Security is based on the assumption that no other third party is able to guess this symmetric key.</li>
</ol>
<!-- -->
<ol>
<li>The sending party is encrypting all further data with the symmetric key; sometimes they swap the symmetric key. Note that the symmetric key is sent by means of the initial message exchange, that means it is encrypted with the public key of the sending party - so the trust that we put into the initial message exchange is the basis for the trust that we place in the security of all further communications.</li>
</ol>
<!-- -->
<ol>
<li>This two-step schema is used in <a href="http://en.wikipedia.org/wiki/Transport_Layer_Security">SSL/TLS</a> , <a href="http://en.wikipedia.org/wiki/Pretty_Good_Privacy">PGP</a> , probably all encryption methods are using this in some way.</li>
</ol>
<p>The catch is that the whole schema depends on the way that the symmetric key is picked, normally a random number generator is supposed to do this; Security is compromised if a third party can guess the random numbers that were picked; Many attacks on privacy are based on manipulating the way that random numbers are chosen on the computer of the sending party;
A somewhat more secure approach would be to encrypt the whole message with public key, and to decrypt it by using the public key - without having access to the private key it is currently impossible to decrypt the message (provided that the keys are sufficiently ‘strong’ - the longer the key, the stronger it is and the harder it is to break by guessing the right combination - one has to try more combinations to be lucky)
The secret of the private key is generated only once, and everything is secure if the computer was not compromised at the time that they key was created.</p>
<p>The problem with this approach is that this takes a lot of time to decrypt the message and slightly less time to encrypt the message; however modern computers have an increasing number of processors. This little project explores how multi processing can help to speed up the encryption and decryption of larger messages.</p>
<p>Another thing is that while it is currently practically impossible to break RSA encryption with reasonably strong key, theoretically this might change once they have a working quantum computer; However there are <a href="http://www.scottaaronson.com/blog/">serious doubts</a> that we are anywhere near the development of such a breakthrough, but things might change, you never know.</p>
<h3 id="the-tools-used-in-this-study-and-how-to-use-them">The tools used in this study and how to use them</h3>
<p>For this study command line programs were written that encrypts/sign and decrypt/verify longer data all with a public/private key pair.</p>
<p>First of all the openssl utility is used to create the key pair; the next command creates the super secret private key - with 4096 bits long. Remember that the longer the key, the harder it is to break, 4096 bits is very secure by any standards (of today).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> openssl genrsa 4096 >private.pem
</code></pre></div></div>
<p>(if on Linux: before doing this move around the mouse in random patterns and type some gibberish on the keyboard; the mouse driver and keyboard driver are some of the sources of entropy used to seed the random number generator used to compute the keys <a href="http://en.wikipedia.org/wiki/Entropy_(computing)#Linux_kernel">here</a> )</p>
<p>Given this file we extract the public key. the sender of the message must have it.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> openssl rsa -in private.pem -out public.pem -pubout
</code></pre></div></div>
<p>The message is encrypted and signed with the following command, in this example the file raw_message is encrypted by mean of the public key public.pem, resulting in encrypted file encrypted_file.file . Note that this program splits up the work so that it can be done in parallel.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> rsaencmt -k public.pem -i raw_message.file -o encrypted_file.file
</code></pre></div></div>
<p>The encrypted message is later decrypted and verified with the following example,
Note that this program splits up the work so that it can be done in parallel.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> rsadecmt -k private.pem -i encrypted_file.file -o decrypted_file.file
</code></pre></div></div>
<p>Also two programs where written that do all the work serially (not in parallell). these are not very practial, they only exist as a baseline in order to compare the times.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> rsaenc -k public.pem -i raw_message.file -o encrypted_file.file
rsadec -k private.pem -i encrypted_file.file -o decrypted_file.file
</code></pre></div></div>
<h3 id="is-it-any-good">is it any good?</h3>
<p>The <a href="http://en.wikipedia.org/wiki/RSA_(cryptosystem)">RSA algorithm</a> is for this utility because there are <a href="http://www.theguardian.com/world/2013/sep/05/nsa-how-to-remain-secure-surveillance">some doubts</a> in the validity of elliptic curve cryptography (given all those recent revelations); also they seem to know how to break at least some variants of AES cipher (<a href="http://www.spiegel.de/international/germany/inside-the-nsa-s-war-on-internet-security-a-1010361.html">here</a> )</p>
<p>However RSA has problems of its own: there are known side-channel attacks; if the computer that is encrypting the message has been compromised then you can break RSA encryption if you can measure fluctuations of the power consumption while encrypting/decrypting; Still this program does make side channel attacks more difficult: encryption/decryption is delegated to multiple cores on the system, to the best of my knowledge it is impossible to measure the power consumption of individual hardware cores/hardware threads.</p>
<p>Maybe for practical encryption systems you would better use several encryption step, each step is applied on the input of the previous step. Each encryption step would use a different symmetric cipher with its own key. The keys would be still encrypted with the some kind of asymmetric encryption like RSA - so you still have the problem of possible side channel attacks.</p>
<p>Chaining of several the following advantage: if at least one of the steps has not been compromised then the end result is still safe (given that the encryption keys have not been compromised), also it is harder to apply side channel attacks if all steps are made concurrently - because it is harder to measuring the time of the encryption steps.</p>
<p>the tool described in this text does not do this kind of chaining.</p>
<h3 id="how-to-download-the-sources--build-the-tools">how to download the sources & build the tools</h3>
<p>Sources can be downloaded as <a href="https://github.com/MoserMichael/cstuff/zipball/master">zip archive</a>
or downloaded by the git program.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> git clone https://github.com/MoserMichael/cstuff.git
</code></pre></div></div>
<p>The project can be build on Linux (checked 32 or 64 bit Intel/Amd) or on Windows with Cygwin.
In order to build the program, we need the presence of an internet connection, so that the build procedure can download the openssl libraries (alternatively you can place the openssl tar ball into directory cstuff/rsautl/openssl</p>
<p>Build and install into /usr/local/ directory</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> cd build
./build-rsautl.sh install
</code></pre></div></div>
<p>Build and install into /usr/alt directory</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> cd build
` ./build-rsautl.sh install INSTALL_PREFIX=/usr/alt
</code></pre></div></div>
<h3 id="how-it-works--file-formats">How it works & file formats</h3>
<h4 id="task-of-the-encryption-program">Task of the encryption program</h4>
<ul>
<li>A header is put before the input message, The header is of the following form
<ul>
<li>4 bytes - magic constant (0xABCDEF)</li>
<li>4 bytes - file version (currently 0x1)</li>
<li>4 bytes - length of input data</li>
</ul>
</li>
<li>The input message is divided into smaller blocks, each block is equal to the size of the public key, minus length of cryptographic padding.</li>
<li>The each block is RSA encrypted with the public key, while using the PKCS1_OAEP cryptographic padding</li>
<li>a SHA256 digest is computer over all encrypted output, this digest is used to verify that the encrypted message has not been altered in transition</li>
<li>the digest itself is encrypted with the public key</li>
</ul>
<h4 id="task-of-the-decryption-program">Task of the decryption program</h4>
<ul>
<li>The first block is decrypted with the private key
<ul>
<li>verify the file header; if magic constant and file version do not make sense, then this means that the message was encrypted with a non matching public key.</li>
<li>check that the length of the file is consistent with the length field of the file header.</li>
</ul>
</li>
<li>Break up the input message into chunks equal to the length of the private key; compute the SHA256 digest over the encrypted data (prior to decrypting)</li>
<li>Decrypt each data chunk using the private key, while using the PKCAS1_OAEP cryptographic padding</li>
<li>when finished with the data: compare the value of the SHA256 digest with the stored value; if it is not equal then fail (show error message)</li>
<li>Strip the header from the decrypted message</li>
</ul>
<h3 id="more-internals">More internals</h3>
<p>The program uses the <a href="http://www.openssl.org/">openssl library</a> for cryptography. The openssl tarball includes instructions on working with multiple threads (File doc\crypto\threads.pod and directory crypto\threads have a sample)
Basically the program needs to install an optional callback function that implements locking for sensitive data structures.</p>
<p>The <strong>rsaencmt</strong> and <strong>rsadecmt</strong> programs are doing their task in parallel; In order to do so they spawn a number of thread - their number is equal to the number of processor cores on the system times 2.
The factor of two is chosen due to hyperthreading - usually a core has two hardware threads, if a given core is stalled then the second thread has the chance to do something useful. Currently I do not set the processor affinity of each thread, as I understand the CPU scheduler of the operating system does its own tricks, and tuning of CPU affinity can be tricky undertaking.
The <a href="http://mosermichael.github.io/cstuff/docs/html/tutils/a00014.html">tpool thread pool</a> is used to dispatch requests to a thread pool with fixed number of threads.</p>
<p>At first the multithreaded program did not perform very much faster than the serial program; as it turns out this was due to locks used by the openssl library.
However with the openssl library it is relatively easy to fix problems related to locking, a callback function for locking must be implemented and registered by means of the <a href="http://www.openssl.org/docs/crypto/threads.html">CRYPTO_set_locking_callback</a> function.
It turns out that each time that there were multiple locks when calling <a href="https://www.openssl.org/docs/crypto/RSA_public_encrypt.html">RSA_public_encrypt</a> and <a href="https://www.openssl.org/docs/crypto/RSA_public_encrypt.html">RSA_private_decrypt</a>
Turns out that the library performs a trick called <a href="http://en.wikipedia.org/wiki/Montgomery_reduction">Montgomery reduction</a> (<a href="https://www.openssl.org/docs/crypto/BN_mod_mul_montgomery.html">this function</a> )
intermedeate values have to be cached somewhere - the openssl library uses the RSA key object for that, now in order to make this library usable with multithreading, every access to these cached values must be locked !
The solution was to</p>
<ul>
<li>Disable this particular lock while building openssl for this project</li>
<li>for each thread create a copy off the RSA key object.</li>
</ul>
<p>However there is no good fix for <a href="https://www.openssl.org/docs/crypto/RSA_public_encrypt.html">RSA_public_encrypt</a> . Remember that <a href="http://en.wikipedia.org/wiki/Padding_(cryptography)">cryptographic padding</a> is used for each block,
the library must create random numbers for each encrypted block, in to do so it must it uses the single instance of the <a href="http://www.cypherpunks.to/~peter/06_random.pdf">SSLeay random number generator</a> , this is a cryptographically secure random number generator (<a href="http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator">more here</a> )
I can’t create a random number generator per thread because that could deplete the entropy pool and seeding of the generator may block for some time !)</p>
<p>Therefore multithreaded encryption is often not very much faster than single threaded encryption, but the speedup for decryption (where it counts) is considerable.</p>
<h3 id="results">Results</h3>
<p>The test encrypts and decrypts a 100kb file with keys of various length; here are the results</p>
<h4 id="laptop-one-processor-with-eight-cores--intel-core-i7-2720qm---220-ghz--on-windows-under-cygwin---program-is-32-bit">Laptop, one processor with eight cores | (intel Core i7-2720QM - 2.20 GHZ); | on Windows under Cygwin - program is 32 bit</h4>
<p>Key length: 2048 bits
|Mode|Encryption|Decryption|
|Single thread|0.154|3.463|
|Multithreaded|0.116|0.850|</p>
<p>Key length: 4096 bits
|Mode|Encryption|Decryption|
|Single thread|0.347 sec|11.905 sec|
|Multithreaded|0.287 sec|3.0222 sec|</p>
<h4 id="eight-cores-two-cpus--intel-xeon-cpu-e5-2609-0--240ghz--linux-on-x86_64">Eight cores (two cpu’s) | Intel® Xeon® CPU E5-2609 0 @ 2.40GHz | (Linux on x86_64)</h4>
<p>Key length: 2048 bits
|Mode|Encryption|Decryption|
|Single thread|0.038|0.898|
|Multithreaded|0.013|0.115|</p>
<p>Key length: 4096 bits
|Mode|Encryption|Decryption|
|Single thread|0.053|2.936|
|Multithreaded|0.017|0.382|</p>
<h4 id="24-cores-2-cpus--intel-xeon-cpu-e5-2630-0--230ghz--linux-on-x86_64">24 cores (2 cpus) | Intel® Xeon® CPU E5-2630 0 @ 2.30GHz | (Linux on x86_64)</h4>
<p>Key length: 2048 bits
|Mode|Encryption|Decryption|
|Single thread|0.042|0.790|
|Multithreaded|0.023|0.087|</p>
<p>Key length: 4096 bits
|Mode|Encryption|Decryption|
|Single thread|0.057|2.527|
|Multithreaded|0.024|0.223|</p>
<h4 id="on-vmware-virtual-machine-that-utilizes-two-core-linux-on-i686">On VMWare virtual machine that utilizes two core (Linux on i686)</h4>
<p>Key length: 2048 bits</p>
<table>
<tbody>
<tr>
<td>Mode</td>
<td>Encryption</td>
<td>Decryption</td>
</tr>
<tr>
<td>Single thread</td>
<td>0.573</td>
<td>14.645</td>
</tr>
<tr>
<td>Multithreaded</td>
<td>0.521</td>
<td>9.297</td>
</tr>
</tbody>
</table>
<p>Key length: 4096 bits
|Mode|Encryption|Decryption|
|Single thread|0.931|53.61|
|Multithreaded|0.897|29.773|</p>
<h4 id="summary-of-results--more-internals">Summary of results / more internals</h4>
<p>Results are consistent with predictions. Encrypting everything with RSA may still not be good enough for bulk data that spans many megabytes, but for shorter text it seems good enough.</p>
<p>Now I could have achieved a greater speedup if I were to use the GPU of the graphics card, but this a bit too complex for an evening of work.</p>
<h4 id="epilogue">Epilogue</h4>
<p>Looking at the details of cryptography one can see to what extent technology did progress since the end of World War II.
The <a href="http://en.wikipedia.org/wiki/Enigma_machine">Enigma</a> and <a href="http://en.wikipedia.org/wiki/Lorenz_cipher">Lorenz</a> machines used electrio-mechanical tricks, several hundreds of moving parts, in its complexity that was very similar to the typewriter that my father was using back when I was a kid.
Now compare this complexity to the number of circuits in a modern computer. Amazing, Science fiction ! Also remember that back then everything related to cryptography was very tightly regulated.</p>
<p>… and the only way to break it is by cheating with the random number generator, by means of <a href="http://en.wikipedia.org/wiki/Side_channel_attack">side channel attacks</a> (such as watching <a href="http://en.wikipedia.org/wiki/Differential_power_analysis#Differential_power_analysis">power consumption</a> directly ) ; or by the mythical beast - the quantum computer.</p>
The Russian state and its evolution
2014-01-19T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2014/01/19/su-kaputt
["", "ramblings"] | <notextile>The Russian state and its evolution</notextile>
<hr />
<p class="publish_date">
19 January 2014
</p>
<p>`Yegor Gaidar: “Authority and Ownership, the Troubles vs the Institutions; state and its evolution.”</p>
<p>This interesting book can be read for <a href="http://lib.ru/POLITOLOG/GAYDAR_E/vlast_sobstvennost.txt">free in Russian</a>, written by <a href="http://en.wikipedia.org/wiki/Yegor_Gaidar">Yegor Gaidar</a> who was the Prime minister of Russia during much of 1992; most works on history by former politicians are a bit apologetic, to some degree this is can be said about this work too.</p>
<p>The first part of the book is an interesting study on the subject of collapsing centralized states; it is less about the causes that cause the collapse of the old order, more about the process of disintegration once the old order has crashed. The crucial problem is that of feeding the cities: disrupted institutions of the state make it difficult to produce and distribute food; once this happens the new powers usually start to expropriate grain by force, this in turn leads directly to civil war and terror; once order returns it turns out that the new state is surprisingly similar to the old order of things.</p>
<p>The author might have intended to make the point that his own policies of shock therapy contributed to having saved Russia from civil war; I understand that another point is that during 1992 there was no time left, no organizational means for a more gradual <a href="http://en.wikipedia.org/wiki/500_Days">500 days program</a> . Besides that the ruling class was busy with silently looting/privatizing all remaining assets.</p>
<p>The study is interesting in in the way that historical phenomena are explained by economic factors; I guess that such a reading of history is the only positive legacy of Marxist thinking. Another interesting theme is the influence of economic thinking upon the policy of the day; during 1917 the prevalent opinion was that a planned economy is a good thing, by 1989 the opposite was held as true; (also see <a href="http://mosermichael.github.io/cstuff/all/ramblings/2013/03/20/post-61.html">my article on slavery</a> , and how the shift to laissez fair economics played a role in the abolition of slavery)</p>
<p>The second part of the book explains the resulting order in historical context: the surprising result of the <a href="http://en.wikipedia.org/wiki/Perestroika">Perestroika</a> is that we saw the peaceful transformation of the Soviet <a href="http://en.wikipedia.org/wiki/Nomenklatura">Nomenklatura</a> (also known as <a href="http://en.wikipedia.org/wiki/New_class">new class</a> ,<a href="http://en.wikipedia.org/wiki/State_socialism">etc</a> ,<a href="http://en.wikipedia.org/wiki/Degenerated_workers'_state">etc</a>) into capitalist managers and owners; the political changes that legitimized the big switch were brought about by those opposing the rulers - Gaidar headed the government of Democrats that brought change (this as a result buried the democrats for good - very few have voted for them ever since); So he needs to do some explaining;</p>
<p>This explaining he does that quite effectively; his analysis is based on Marx’s theory of the <a href="http://en.wikipedia.org/wiki/Asiatic_mode_of_production">Asiatic mode of production</a> - unlike other parts of the world, Western Europe developed the unique and extremely effective notion of private property; (actually this work by Marx was censored during Stalin). Production in the west is based on private ownership and the rule of law; in other areas of the world the Ruler/owner would grant fiefdoms to his vassals, so property and power were inseparable. Gaidar describes the evolution of the Russian state in terms of this concept; the history of Imperial Russia and Soviet Union as two instances of the Asiatic mode of production. The reforms of the 90ies are portrayed as an attempt to buy out sRussia from the Nomenklatura; the rulers would turninto legitimate owners in turn for concession that would establish a free market and the dominance of civil society. (well I guess the plan did not quite work out, at least until now - they rulers had their problems with the ceding power part, they did not quite do it)</p>
<p>I think this book has some wider importance besides the context of Russia, first of all it covers problems that other countries are facing, I guess China is facing very similar problems. Also the Western world also knows some formulas that socialize losses while profits are privatized by the manager elite.
I think the problem of a looting manager classes is not limited to the disintegrating Soviet Union, its the opportunity that makes the thieve. Opportunities for large scale theft arise when the managers oversee systems that are far too complex for effective oversight; complexity makes effective oversight difficult when the issues are not clearly understood by those bodies that are supposed to create the ‘checks and balances’; The counter force against this tendency is a better dissemination of knowledge/information, this has been very much aided by the Internet, therefore the internet might gradually turn into a real check of those in power; lets hope that this institution will not be curtailed in novel ways - as might happen in the name of fighting terrorist, enforcement of copyright laws or precaution out against other nastiness (watch out for the hidden motive).</p>
<h3 id="summary-of-reasons-for-the-collapse-of-the-soviet-union">summary of reasons for the collapse of the Soviet union</h3>
<p>The good part that it is much shorter than the “The History of the Decline and Fall of the Roman Empire”</p>
<ul>
<li>
<p>Soviet manager class (Nomenklatura): After Stalin the manager class no longer feared repressions; they formed many informal relations among their own kind and everything was decided by means of an internal bureaucratic barter market; The manager class was also aspiring to consolidate its status; they internalized their own position as real masters of the show. In the end they wanted to appropriate state owned property and turn it into their own private property; now Perestroika created the opportunity to to do just that: new private cooperatives made insider deals with management, these deals enabled large scale theft/privatization of state owned assets; actually the same scheme was in widespread use during the 1920ies - during the NEP period. Crumbling administrative controls during Perestroika were of much help during the late 1980ies.</p>
</li>
<li>
<p>collectivized agriculture was inherently inefficient - farmers just had absolutely no incentive to work.
In the 70ies-80ies they tried to fix this by technological means: mountains of fertilizers, thousands of tractors, enormous amelioration works - all in vain. By the 80ies the USSR was a heavily urbanized country, so the trick of just dismantling state owned farms would not work here; there was nobody left who would be ready to work the land (however in China this step was quite successful).</p>
</li>
<li>
<p>They had to sell oil in order to buy grain; by 1985 oil prices went down, at the same time salaries went up; by 1990 the country was heavily in debt and nobody was ready to give them additional loans.</p>
</li>
<li>
<p>most of industry was working to produce weapons for the military; consumer goods were of a distant second priority. By the end of the 80ies they were producing mostly junk that could not be exported.</p>
</li>
<li>belated attempts of reform were not very effective: from 1985-87 it was about fixing ‘isolated shortcomings’ ; from 1987-89 reforms were about ‘democratic socialism’; from 1990 onward they talked about transition to free market economics.
At the same time administrative controls crumbled, the system was based on fear, once that element was gone there was nothing to hold/bind things together; with the absence of fear the administrative system crumbled.</li>
<li>the break up became inevitable after the armed coup of august 1991 : all republics just wanted out as a measure of precaution; the question of safety was very important, nobody knew would would come next in such a dysfunctional state; also the system just disintegrated. A power vacuum appeared, so local republics just took over.</li>
</ul>
<h3 id="did-the-soviet-union-collapse">did the soviet union collapse?</h3>
<p>It is sometimes argued that the western Roman Empire did not collapse and that it gradually transitioned into the feudal order of the middle ages, maybe that’s what’s happened to the Soviet union?</p>
<p>What speaks for this argument is the fact the continuity of leadership - the fact that the nomenklatura just happened to privatize its assets and continue to rule under the new terms.</p>
<p>Anyway the democratic transformations after the collapse of the Soviet Union were not that very permanent, for example they failed to produce an independent judiciary, so that there was not deep rooted separation of powers.</p>
<p>Maybe the whole transformation into a democratic Russia was just a temporary re-branding, which was anyway followed by a later return to autocratic rule under Putin.</p>
<h3 id="could-things-have-turned-out-different-by-adopting-a-more-gradual-program-of-economic-liberalization-at-an-earlier-stage-">Could things have turned out different by adopting a more gradual program of economic liberalization at an earlier stage ?</h3>
<p>small scale private enterprise was allowed by the <a href="http://www.encyclopedia.com/doc/1G2-3404100303.html">law of cooperatives</a> since May 1988, Gaydar explains that cooperatives were an important tool that enabled the looting of state owned assets by the Nomenklatura. Still i wonder what would have happened if they had introduced this measure in 1985, when the process of disintegration was less advanced;</p>
<p>The reforms in China were driven by decentralization/de-collectivization of agriculture; this could not have worked in the Soviet Union for a variety of reasons: it is argued that the initiative required of independent farmers was being suppressed for decades and would not just have appeared overnight; agriculture was more industrialized/reliant on machines (self reliance small farms were harder to implement); also farmers got used/were content with just slacking off.</p>
<p>However what might have been done is to allot larger plots of land to peasants for private use <a href="https://ru.wikipedia.org/wiki/%D0%9F%D0%BE%D0%B4%D1%81%D0%BE%D0%B1%D0%BD%D0%BE%D0%B5_%D1%85%D0%BE">here</a>
%D0%B7%D1%8F%D0%B9%D1%81%D1%82%D0%B2%D0%BE <a href="http://wiki.ru/encyclopedia/%EF%F0%E8%F3%F1%E0%E4%E5%E1%ED%FB%E9+%F3%F7%E0%F1%F2%EE%EA/">here</a> - that was one of the more effective measures of <a href="http://www.globalsecurity.org/military/world/europe/hu-history-33.htm">Goulash communism</a> ;</p>
<p>No way to know would could have been ?</p>
<hr />
<h3 id="yegor-gaidar-authority-and-ownership-the-troubles-vs-the-institutions-state-and-its-evolution">`Yegor Gaidar: “Authority and Ownership, the Troubles vs the Institutions; state and its evolution.”</h3>
<p>Notes that I took while reading <a href="http://lib.ru/POLITOLOG/GAYDAR_E/vlast_sobstvennost.txt">this book</a>,</p>
<h3 id="first-part-the-troubles-vs-the-institutions">First part: ‘the Troubles vs the Institutions’</h3>
<p>Agrarian societies (until late 18th century) is the dominant forms of political organization</p>
<p>Feudal states, common features - decentralized state + no standing army + few taxes by the crown/center (the local lord is the Boss)
* Problems: potential for internal strife + weak army
* Advantages: less complex state has less potential for system crash;
Tradition aids/legitimizes the power relationship between Lord and Serf
(in times of crisis the power relationship is not questioned; - it has always been like this -)
There are lots of crisis situations, but system does not break down as a whole.</p>
<h4 id="centralized-states">Centralized states</h4>
<p># one tax authority for the whole state
# complex bureaucracy ; roads/post/communication system required for tax levies ; conflicts resolved by legal system and a hierarchy of courts
# a standing army financed from taxes
# food reserves/gold reserves to bridge situations of crisis / this increases stability of the system</p>
<p>* Problem: complex organization has serious potential for breakdown of state (what happens if taxes base fails and no one can feed the bureaucracy ?)
* Far away local bureaucrats are hard to oversea from the center.
* Stability is a constant search for equilibrium between taking away from the serfs without driving them into ruin;
* Threat of invasion can lead to over taxation; tax based is diminished by ruined serfs; resulting in crash of the system;
* As tax base is breaking down, the local bureaucrats turn to requisition by force; conflicts sharpen
* troubles lead to total breakdown of law and order (as a first step criminals freed from prison): The state has no monopoly on the use of force; there is no Lord/Sheriff to take over the task;
* if there are complex irrigation systems then these fall into disrepair / Finance system , long distant trade - all gone.
* State structures vanish ; more basic relationships last longer: family relationships; master->servant/serf relationship</p>
<h4 id="modern-revolutions">Modern revolutions:</h4>
<p><a href="http://en.wikipedia.org/wiki/English_Civil_War">English civil War</a> / <a href="http://en.wikipedia.org/wiki/Dutch_Revolt">Dutch revolt</a> / <a href="http://en.wikipedia.org/wiki/American_Revolution">American revolution</a> - did not break with tradition; these revolutions did not sweep away the courts/finances/taxes did not crash;</p>
<p>French revolution was a different thing - it was a total break with past traditions</p>
<p>* French state before revolution had high degree of stability; no external threats;
* problems: financial instability/debts; failed harvest/famine of 1788 (problems after a failed harvest become apparent the next year)
* 1789: becomes clear that army does not follow orders (neither of the old or the new regime); the old order crashed within days;
* 1791: taxes abolished (<a href="http://en.wikipedia.org/wiki/Taille">Taille</a> tax) ; (along with the whole old order) ;
* New order equality before the law, abolishes <a href="http://en.wikipedia.org/wiki/Estates_of_the_realm">Estates of the realm</a> ; guarantees of ownership; freedom of expression; participation; secular state (problem: new order not legitimized by tradition, no means to levy taxes as the old system had to be dismantled)</p>
<p>* State need to finance its activities: requisition of church estates; these were cheap because of the revolution: there were few guarantees of ownership.
* Printing presses emit papers; inflation sets in;
* farmers have no incentive to sell grain - wait for better prices; states start requisitions by force.
* System of laws was enacted - requisition by force sets in
<strong>** 4/5/1793 “loi du maximus” - regulates grain trade; price controls; supervisors present at markets
**</strong> 15/8/1793 grain tax;
<strong>** 1/10/1793 <a href="http://en.wikipedia.org/wiki/General_maximum">General Maximus</a> fixes prices. enforcement of law now by means of terror; <a href="http://en.wikipedia.org/wiki/Law_of_Suspects">laws of suspect</a> !
**</strong> 1794-1795 Food scarcity & Hunger; the Terror was also a war against the farmers/result of requisitions by force.</p>
<p>Difference between troubles and revolution - when revolution ends, the main changes are taken over by the new order. Napoleon and later under Restoration preserved equality before law and property relations;</p>
<h4 id="generalization">generalization</h4>
<ul>
<li>Civilization is a fragile construct; it can’t be maintained when cities are not fed and law and order is not maintained;
In olden times consumption/production of food was decentralized - no need for hierarchies, now society is more complex: example electricity - in winter a few weeks without it is a major calamity.</li>
<li>crucial component of stability - the states monopoly on violence.</li>
<li>Army is part of society - if there is a general feeling that powers are corrupt or illegitimate then this talk reaches the armed forces;
Any sign of disobedience is a mortal danger to any regime; if the armed forces refuse to shoot at demonstrators then the regime is finished - but this also finishes the army; army disintegrates: if orders are questioned and not followed then there is no army. The streets fill up with armed and aggressive young men.</li>
<li>Police paralysis: after a revolution the prisoners are all freed wholesale - when prisons are stormed then no one will be able to distinguish between criminals and political prisoners.
Police does not work without trust/cooperation of population; Police can’t work when trust this is lacking, police officers are laid off or killed, police informers no longer cooperate.</li>
<li>After the revolution competing factions turn into competing centers of power - each one wants to be the real power. People question authority; this is the direct result of the financial crisis and the inability to distribute food.</li>
<li>Property - social relationship that relies on functioning courts and state; agricultural production severely affected by lack of clearness in matters of property question.</li>
<li>Taxation: no taxation after revolt, functioning of the state is now financed by means of printing money - inflation sets in; When trust in money is destroyed the state just stops payments of salaries.</li>
<li>Rural areas are differently affected: transport system is down + lack of property relationship + worthless money + now need to pay taxes all lead to depression in agricultural output; Farming becomes subsistence farming.</li>
<li>It becomes a challenge to feed the cities: the masses that caused the overthrow of the old system start to grumble, the new order must buy loyalty of its power base. Result: decision to appropriate food by means of requisition leads to war of the city against the peasant. Civil war starts.</li>
</ul>
<h4 id="russian-revolution">Russian revolution</h4>
<ul>
<li>Prior settings: Most lands owned by aristocracy; Stolypin reforms created a new class of farmers/owners; by 1914 most of the grain is produced by them - subsistence farming still very widespread. Muzhiki/peasants are very hostile to the alien influence of the city; “Miroed” - the one who is against the collective.</li>
</ul>
<!-- -->
<ul>
<li>1914: Russia enters war, budget still balanced and strong gold reserves; still the war had to be financed by the printing press; 14-17 Russia - money mass in circulation increases x6; (France x2, Germany x3).</li>
<li>Farmers reluctant to sell grain due to inflation; 1916 - government makes plans for price fixing + requisitions! but is already not able to implement these measures by force.</li>
</ul>
<p>Like previous revolutions/revolts - its the outbreak was totally unexpected by contemporaries;</p>
<ul>
<li>start of disorders: due to lack of transportation shortages lead to bread lines in Petrograd (all trains busy in the 1st world war); still bread was available!</li>
<li>protests were not against the war - many workers in Petrograd worked in armament factories and were not against the war.</li>
<li>
<h2 id="2621917-army-refuses-to-shoot-when-protests-reached-a-critical-mass---the-regime-is-finished-within-24-hours">26/2/1917 Army refuses to shoot when protests reached a critical mass - the regime is finished within 24 hours.</h2>
</li>
<li>Law and order: 27/2/1917 All prisoners freed. 4 March: Old police is disbanded (provisional government had to appease the Petrograd Soviet - the institution that controlled the garrison), new militia formed on paper only - until late 1918</li>
<li>by June 1917 - peasants take over / destroy the landowners estates; no formal property relations exist - land being constantly redistributed, this chaos was a further disincentive to till the land.</li>
<li>Bread: initially tried to continue price fixing + requisition, was politically impossible, also could not work due to lack of state bureaucracy (regional authorities disbanded)
Aug-Sept 1917 - raises purchase price x2 ! This measure works for the time begin, temorarily.</li>
<li>complete disintegration of army takes several months - new powers are distrustful of it (order #1 - capital punishment abolished); still discipline and order are not completely gone with the ancient regime.</li>
<li>
<h2 id="printing-presses-are-only-mean-to-finance-state-prices-rise-x3-till-end-of-1917">printing presses are only mean to finance state: prices rise x3 till end of 1917.</h2>
</li>
<li>July,Oct 1917 - in this vacuum a small number of armed men was sufficient to take power at the center. Two entities did not dissolve - the <a href="http://en.wikipedia.org/wiki/Czechoslovak_Legion">Czechoslovak Legion</a> and <a href="http://en.wikipedia.org/wiki/Latvian_Riflemen">Latvian Riflemen</a> - these were made up of foreigners. this was the real Red Army until vacuum ended by the fall of 1918</li>
<li>Bread:
<ul>
<li>First measure was requisition from big traders ; initially tried to organize barter for food - impossible due to lack of organization/transport.</li>
<li>Reds believe in centralized planning (example of Germany in WW1) - did not try buy grain for market prices. the old state had to be completely destroyed;</li>
<li>Requisition: land owners estates were ruined, still existed the big farmers; Reds tried to create power base among poor peasants - divide and conquer (failed, at this moment there was less inequality to exploit than before)
Requisitions by force lead directly into civil war; still total failure in May-July 1918 due to lack of reliable armed forces / transport is in chaos / no way to define norms by territory - no management/bureaucracy, without that requisition is just armed robbery.
Summer 1918 - meetings and strikes demand free trade / closing of barrier troops;
July 1918 - temporary dismantling of measures: barter trading allowed, the state buys up grain and raises purchase price! same steps taken by provisional government earlier
Aug 1918 - firm that sends workers to barrier troops gets 50% share of the loot; this incentive motivates many to join (previous had problem to recruit for this task)</li>
<li>Only way to survive is individual barter with farmers; Barrier troops were supposed to stop this, but these did not quite follow orders.</li>
</ul>
</li>
</ul>
<!-- -->
<ul>
<li>
<h2 id="august-1918---red-terror-the-cheka-starts-its-killings--interesting-the-author-does-not-mentionomits-this-fact-autumn-1918---the-red-army-is-formed-on-basis-of-discipline-the-soviet-state-as-we-know-it-exists">August 1918 - red terror; the <a href="http://en.wikipedia.org/wiki/Cheka">Cheka</a> starts its killings (? interesting: the author does not mention/omits this fact); Autumn 1918 - the <a href="http://en.wikipedia.org/wiki/Red_Army">Red Army</a> is formed on basis of discipline; the Soviet state as we know it exists.</h2>
</li>
<li>Bread: start 1919 - each village would be given norms and ‘collective responsibility’ was enforced (Meet the new boss. <a href="http://www.youtube.com/watch?v=Rp6-wG5LLqE">Same as the old boss</a> ) - the village would have to enforce the norms, else …. expropriations are effective; but also Red Army is much larger and has priority over the city - result: cities still hungry</li>
<li>Peasants in conflict - don’t like the expropriations by the Reds, but they want to keep changed property relations - are afraid that the whites will roll back to the ancient regime.
Result: the Reds win - they were regarded as the lesser evil.
Still: area that is tilled decreases - due to disincentive of expropriations and disrepair of agricultural equipment.</li>
<li>The Reds offered minorities their own republics (with formal right of secession); this did win them over.
(also: the Poles offered neutrality in return - this allowed the Reds to move forces at critical moment of the civil war)</li>
</ul>
<h4 id="breakup-of-the-soviet-union">Breakup of the Soviet Union</h4>
<p>Sources of the crisis:</p>
<ul>
<li>1930-1985 agriculture based on collectivized farms; expropriation of grain by force - all in order to enable/finance shock industrialization + heavy industry buildup. System is not efficient at producing - urbanization proceeds and more people must be fed ; as a result they must buy more grain on world market for dollars.</li>
<li>1970 - growth increasingly dependent on export of oil/later gas.</li>
<li>1985 - energy prices fell sharply in 1985 - in order to bye grain the USSR dependents on loans, debts grow.</li>
<li>1986-1991 - salaries where raised, while production output fell; money went into savings, as there was nothing left to bye.</li>
<li>1989-1991 - creeping privatization, the Nomenklatura takes what it can get, uses the same old schema as during NEP: cooperatives make preferential deals with state owned enterprises, these deals are used to pump assets into private hands.</li>
</ul>
<hr />
<p>22/8/1991 - end of the coup; a power vacuum forms:</p>
<iframe width="420" height="315" src="//www.youtube.com/embed/Eqcyt0Gwhjw" frameborder="0" allowfullscreen=""></iframe>
<ul>
<li>chaos did not lead to violence (<a href="http://www.youtube.com/watch?v=Eqcyt0Gwhjw">as anticipated here</a> ) why? - higher level of education/urbanization; less potential for violence as events did no unfold as a result of a major war; the deterring example of recent Soviet history; responsible leaders who tried hard to avoid violence. (i think the presences of Nukes also had a sobering effect on the leadership, Yugoslavia did not have that component)</li>
<li>Power vacuum: who is in charge - Russia, Republics, the USSR Supreme Soviet ?</li>
<li>in Army: who controls army in breakaway republics? what about ethnicity of officers/soldiers? What about an officers flat in breakaway republic, should he stay or leave? No functioning army, the officers busy with their own problems, Militia disappears; new internal borders not protected; (? how exactly did Armenia and Azerbaijan fight a war without armies?)</li>
<li>After coup: republics all broke away as precaution against derailed/dangerous Soviet State (not just Baltic republics)</li>
<li>Question of borders and ethnic Russians remaining in republics: big potential for violence, things did not go as in Yugoslavia - the nuclear capabilities had positive effect on tempers; Borders kept as - nuclear arsenal goes to Russia.</li>
<li>Bread: soviet system was based on coercion - could not work when threat of repression was not real; No country would grant credit to a bankrupt country - so can’t buy from Canada. The only possibility is to deregulate prices / print money, in the hope that farmers will sell</li>
<li>Big difference: in 1917 the whole establishment believed in central planning and the virtue of state intervention ; the failings of the Soviet state convinced them in the opposite.</li>
<li>inflation eats away peoples savings: what is the reaction? (Me: I guess its the playbook of the Russian state: democracy in times of crisis in order to establish legitimacy) Effectiveness: would farmers sell produce on conditions of destabilized currency? - still it worked somehow. (Me: what was the relative role of foreign aid ? to what extend did it bridge the gap until harvest of 1992 ? - he gives some figures, but does not elaborate)</li>
<li>initially the Soviet Ruble remained common the currency of former republics: republics print money indiscriminately, so Russia is importing their inflation as well.</li>
<li>foreign debt accumulated by USSR: payment was adjourned till end of 1992; Also Russia got new credits from the West (how much) on the basis of having presented a restructuring plan. Sharp budget cuts were made - education, army, farming subsidies - all cut.</li>
<li><a href="http://en.wikipedia.org/wiki/Boris_Yeltsin">Yeltsin</a> knew that he would have to pay with his popularity, but he was the man and understood the risks/necessity of the measure;
<ul>
<li>it was an unusual decision for a politician to subordinate personal interest to those of his country -
Dual power of president vs parliament: existing constitution did not clearly divide power between executive and legislative branches - so it was not clear who was in charge; Could Yeltsin have announced elections in the fall of 91? this would have put him in conflict with the parliament - such a conflict could have turned into civil war. Instead decided to attempt compromise with parliament on the question of a new constitution.</li>
</ul>
</li>
<li>Mid 1992 - old Ruble has started to work somehow; USSR dismantled without civil war; Nukes were all moved to Russia; main crisis averted.</li>
<li>Opposition consolidates: many regard the loss of soviet empire as a trauma - society accepted soviet ideology as integral part of its empire - unlike Poland, where society regarded the Soviet system as occupation, in Poland growth start to pick up around 1992 (Me: also in Poland the former owners were still around, big difference !)</li>
<li>17/6/1992 - parliament passes budged that increases deficit spending, constant wrangling starts.</li>
<li>25/4/1993 - referendum gives boost to legitimacy of Yeltsin; failed to reach compromise on constitution.</li>
<li>Summer 1993 - new Ruble unbinds currency from other republics.</li>
<li>civil war of 1993: 21/9/1993 - Yeltsin decides to dissolve parliament; fighting starts; still army was loyal to president and shot down the insurrection.</li>
</ul>
<h3 id="second-part-state-and-evolution">Second part: ‘State and evolution’</h3>
<h4 id="asiatic-mode-of-production-marx---private-ownership-is-not-prevalent-source-of-production-property-not-enshrined-by-laws"><a href="http://en.wikipedia.org/wiki/Asiatic_mode_of_production">Asiatic mode of production (Marx)</a> - private ownership is not prevalent source of production, property not enshrined by laws.</h4>
<ul>
<li>Only the State/Ruler owns all the lands; everything subordinate to state/bureaucrats power (rulers are semi-theocratic);</li>
<li>Property is an attribute of power; loss of power implies loss of property. Frequent major redistribution of wealth as a result of politics. Private property is tolerated but is always under control/under threat of expropriation.</li>
<li>Insecurity breeds lack of incentive for production, short term gains generally favored over long term investment; no real growth/development.</li>
<li>Insecurity breeds corruption among Aristocracy/bureaucracy (who try to convert power into tangible benefits)</li>
<li>Change/reforms are usually a tightening of administrative control, without real change.</li>
<li>endless dynastic cycle: State gets stronger; aristocrats/bureaucrats loot/steel and weaken the state; disintegration of state - new rulers take over; everything stays the same.</li>
</ul>
<h4 id="ancient-mode-of-production-in-greece---later-rome">Ancient mode of production in Greece - later Rome</h4>
<ul>
<li>based on private ownership, attributed to <a href="http://en.wikipedia.org/wiki/Solon">Solon (Athens)</a> (6th century BC)</li>
<li>production based on private ownership; privately owned slaves;</li>
<li>rights/property of a citizen guaranteed (or rather the norm is not to violate them) ;</li>
<li>Rulers do not have absolute power (???) ; state as instrument of the <a href="http://en.wikipedia.org/wiki/Polis">polis</a></li>
<li>political crisis does not lead to total redistribution of wealth; Rome adopts this system.</li>
</ul>
<h4 id="succeeding-feudal-order">Succeeding Feudal order</h4>
<ul>
<li>No absolute owner/power over of all land; The Crown/Lords/Manor-commons all have some overlapping claims on the land; Catholic Church is a parallel structure;</li>
<li>No strong states: no total wars of extermination (?) that require mobilization of entire society + standing army (this would create a strong state);</li>
<li>selective adoption/preservation of <a href="http://en.wikipedia.org/wiki/Roman_law">Roman law</a> in Medieval states. Decisions must be legitimized by tradition.</li>
<li>increasingly independent <a href="http://en.wikipedia.org/wiki/Market_town">Market towns</a> - <a href="http://en.wikipedia.org/wiki/Guilds">Guilds</a> and later <a href="http://en.wikipedia.org/wiki/Hanseatic_League">Hansa cities</a> - gain protection against <a href="http://en.wikipedia.org/wiki/Robber_baron">Robber barons</a> ; possible to get rich without having to be near a center of power.</li>
<li>weak state creates room for development of multiple independent structures <a href="http://en.wikipedia.org/wiki/Civil_society">Polis / civil society</a> .</li>
<li>relative stability + guarantees of private ownership create incentive for innovation and growth.</li>
</ul>
<h4 id="meanwhile-in-the-rustsardom-of-russia">Meanwhile in the Rus’/Tsardom of Russia</h4>
<ul>
<li>problem of confronting Nomads: the state as shield ; all of society mobilized into effort</li>
<li>Similar problems at European fringe: Spain vs Moors ; Austria vs Ottomans - all lead to strengthened of the Shield-state; this strengthens the centralized the bureaucracy and slows overall development.</li>
<li>upon winning the war with the Khanate states, the Moscow state turns inwards: free trading Novgorod suppressed/incorporated; tight control over land - Noblemen appointed and disowned from Moscow; cities and Orthodox Church subordinated; Serfdom installed.</li>
<li>territorial expansion into Siberia further strengthens the army + bureaucracy; State encourages Xenophobia and imperial complexes; cult of the State worship is established; ( <a href="http://en.wikipedia.org/wiki/Statolatry">Statolatory</a> ) This is the continuous theme that goes on until our days.</li>
<li>Powerful state suppresses civil society - even when there is no external threat; no guarantees against arbitrariness of the state;</li>
</ul>
<h4 id="western-europe-late-medieval-period">Western Europe: Late Medieval period</h4>
<ul>
<li>Strong states appear (Me: in France <a href="http://en.wikipedia.org/wiki/Intendant#History">this happened</a> upon start of 30 year’s war) ; but society is not crushed by the state, legal traditions + social differentiated serve to protect society against arbitrariness.</li>
<li>Mercantilism: the state directly intervenes in the economy (establishment of trading companies, colonial wars, concessions+trading monopolies, protectionism); all this work to strengthen owners - who are independent and not directly subordinate to the state.</li>
</ul>
<h4 id="meanwhile-in-russia">Meanwhile in Russia:</h4>
<ul>
<li>16th century: Russian defeat in <a href="http://en.wikipedia.org/wiki/Livonian_War">Livonian wars</a> make apparent a lag behind the west/need for reforms .</li>
<li>Two options for modernization; <a href="http://en.wikipedia.org/wiki/Peter_the_Great">Peter the Great</a> leans to the first option, but does elements of the second too!
<ul>
<li>Strengthen the state; adopt only technological results while leaving the system as is.</li>
<li>Adopt the western mode of production; Paradox: Peter by force institutes westernized elite that aspires for more independence.</li>
<li>Russian bureaucracy created: combines the worst traits of both East and the West; from Prussia it gets mechanistic approach to/remoteness from people; From the east it gets arbitrariness, unpunctuality and corruptness.</li>
</ul>
</li>
<li>Russia is in permanent catch up race with the west; catching up achieved by directive from above and at great costs;</li>
<li>Subsequent reforms confirm relative independence of landowners (<a href="http://en.wikipedia.org/wiki/Charter_to_the_Gentry">Charter to the gentry</a> among other reforms) turn the elite from keepers into owners of the land ; Problem: serfdom is preserved as is (and manors/commons on the other side too), as a result private ownership of the land never gains wide legitimacy.</li>
<li>Situation becomes permanent - rulers in fear of conspiracies/mutiny by the elite (see Peter III , Pavel I)</li>
</ul>
<h4 id="in-the-west-19th-century-revolutionary-movements">In the west: 19th century Revolutionary movements</h4>
<ul>
<li>19th century - really fast growth + ideals of equality all create great expectations with the poor sections of society;</li>
<li>Source of the crisis: technological development outpaced traditions ; crisis of too high hopes/expectations ; crisis of legitimacy; Seeming contradiction between rationality of production vs the chaos of the market; Anger fueled by frequent busts/crashes; Marx is wrong - productive forces were not superior to the relations of production</li>
<li>discontent is voiced/exploited by intellectuals on the margins who challenge the capitalist order; main threats now are Socialist demagogues.</li>
</ul>
<h4 id="the-socialist-challenge-was-defeated-by-assimilation-in-20th-century-not-suppression">The Socialist challenge was defeated by assimilation (in 20th century), not suppression</h4>
<ul>
<li>Social reform: <a href="http://en.wikipedia.org/wiki/Eduard_Bernstein">Bernstein</a></li>
<li>limited state intervention in economy, while preserving its free market character: <a href="http://en.wikipedia.org/wiki/John_Maynard_Keynes">Keynes</a></li>
</ul>
<h4 id="meanwhile-in-the-russian-empire">Meanwhile in the Russian Empire:</h4>
<ul>
<li>defeat of Russia in Crimean war make apparent a lag behind the west/need for reforms.</li>
<li>Alexander II : 1862 abolition of serfdom, Military reform</li>
<li>Central question: to what degree to allow independence of land ownership/finances/markets from the state</li>
<li>Outcomes:
<ul>
<li>Commons/Community still own land that is not owned by landowner / long term pay-out scheme of commons.</li>
<li>system of collective responsibility for taxation/rents is maintained.</li>
<li>Peasants may not leave without permission of community; may not break away</li>
<li>Egalitarian streak of Russia is preserved; question who owns the land becomes remains critical.</li>
</ul>
</li>
</ul>
<!-- -->
<ul>
<li>1880s finance policies of <a href="http://en.wikipedia.org/wiki/Ivan_Vyshnegradsky">Vyshegradsky</a> ; sound finances and industrialization from above (railroad construction); 1889 return to gold standard.
protectionism to further indigenous production - fast industrial growth; all this is financed by high taxes on peasants - results in political destabilization.</li>
<li>Agriculture: most farmers still on subsistence farming; question: will industry develope and be able to stand on its own feet - or will the taxes spawn political unrest first?</li>
<li>Land question conflicting views: <a href="http://en.wikipedia.org/wiki/Vyacheslav_von_Plehve">Plehve</a> is for keeping control over land; keep peasants in check ; <a href="http://en.wikipedia.org/wiki/Sergei_Witte">Witte</a> says lack of private ownership on land leads to revolution.</li>
<li>Reforms start before 1905 revolution: 1903 - collective responsibility scrapped; 1904 - no corporal punishment; after revolution Stolypin’s reforms.</li>
<li>Revolution of 1905 and defeat in <a href="http://en.wikipedia.org/wiki/Russo-Japanese_War">Russo-Japanes War</a> make apparent a lag behind modernized Japan/need for reforms.</li>
<li><a href="http://en.wikipedia.org/wiki/Stolypin_reform">Stolypins reforms</a> 9/11/1906: allow peasants to break away from manor lands and turn into full owners; encourages resettling to Siberia; Agricultural bank gives loans on land collateral.</li>
<li>reforms start to pick up by 1909; 1909-1913 sees fast growth of industry and agriculture that stands on solid footing.</li>
<li>Stolypin: rare example of Russian statesman who tries to limit the role of the state.</li>
</ul>
<p>from World war I to Communist rules</p>
<ul>
<li>wake of the war: remaining old difference over who should own land not settled - owner peasants still a narrow class (Miroed); rigid bureaucracy + sharp income gap contribute to deepening of political conflitcs;</li>
<li>World war I leads to erosion of morals, legitimacy and social norms; (see <a href="http://archive.org/stream/andquietflowsdon00shol/andquietflowsdon00shol_djvu.txt">And Quiet Flows the Don</a> ) Nobody would understand the goals of the war therefore the regime lost all legitimacy; everyone was against the Zar-Rasputin (Me: why is everyone so against him? Rasputin and Witte were the only ones in Russia who were publicly against entry of Russia into WW I)</li>
<li>Centralized management of economy during the war is the economical model for communism; economics under communism is very militarized and has state as monopolist. Lenin’s work <a href="http://en.wikipedia.org/wiki/Imperialism,_the_Highest_Stage_of_Capitalism">Imperialism as the highest stage of Capitalism</a> as the blueprint for economics of Communism - its describes the nature of the state they built. Monopolies kill free competition resulting into parasitic rent seeking capitalism; lack of incentive for development, resulting in permanent crisis … and that’s exactly what they got in practice!
<ul>
<li>Consensus of the establishment was the central planning (in some form) is a good idea; state control, supervision, accountability were all regraded as good things!</li>
</ul>
</li>
<li>Dictatorship of the party: that’s supposed to make the difference/essence of socialism.
Novelty: the Asiatic mode of production traditionally does not negate the notion of private property; Communism does - this paves the way for total control;
In order to justify total equality of society and state they needed a totalitarian ideology.
(Me: US President <a href="http://en.wikipedia.org/wiki/Calvin_Coolidge">Coolidge</a> nailed it: “government control cannot be divorced from political control” )</li>
</ul>
<h4 id="history-of-nomenklatura-and-the-soviet-state">History of Nomenklatura and the Soviet state</h4>
<p>a short history of the Soviet state:</p>
<ul>
<li>all property is that of the state</li>
<li>state property is collective property of the nomenklatura</li>
<li>Each bureaucrat (and all of them) try to turn collective property into their own property.</li>
</ul>
<h5 id="1920---1930--nep--state-capitalismimperialism-with-elements-of-private-ownership-in-economy">1920 - 1930 : <a href="http://en.wikipedia.org/wiki/New_Economic_Policy">NEP</a> : state capitalism/imperialism with elements of private ownership in economy</h5>
<ul>
<li>What was preventing the Termidor / return to capitalism ?
<ul>
<li>collective fear of bureaucrat: return of old owners was the worst thing that could happen to them; they understood that their own position as a class was not firm.</li>
<li>Repressions within the nomenklatura: dictatorship of the ideology was suppressing the urge to turn state property into their private property.</li>
<li>sectarian nature of Communist ideology:</li>
</ul>
</li>
</ul>
<!-- -->
<ul>
<li><a href="http://en.wikipedia.org/wiki/Partmaximum">Partmaximum</a> - formal limit of managers salary was <a href="http://www.1917.com/History/I-II/rmZH0i30fEkZSDSDxKo7uOw3aNg.html">de facto removed in 1929</a> ; formally abolished in 1932 (by Politburo decision that was kept secret)</li>
<li>During <a href="http://en.wikipedia.org/wiki/New_Economic_Policy">new economic policy</a> there was the wide spread scheme of privatization: official and private industry made deals to channel state resources into private hands; some estimates go that 350 million gold rubles where thus privatized !</li>
<li>Still the nomenklatura craves for political stability and ability to inherit property - private property.</li>
</ul>
<h5 id="1930-1953--totalitarianism--stalinist-terror-individual-is-just-a-grain-of-sand">1930-1953 : totalitarianism / Stalinist terror, individual is just a grain of sand</h5>
<ul>
<li>increasingly nationalist/chauvinist tone of ideology; still internationalist forms are preserved as state incorporates many nationalities (officially equal) and because ideology may not be rocked due to its central role</li>
<li>Ideology looses its appeal; turns into form devoid of substance - first step that system will lead to sobering up of the system.</li>
<li>Increasing income gap between managers and workers.</li>
</ul>
<h5 id="1953-1988--authoritarian-stage">1953-1988 : Authoritarian stage</h5>
<ul>
<li>XX congress proclaims that there will be no return of terror (similar to <a href="http://en.wikipedia.org/wiki/Charter_to_the_Gentry">Charter to the gentry</a> )</li>
<li>change in practice of management; manager form many informal connections; beurocratic barter market forms in process of negotiating/finalizing of production norms ; managers now feel that they are in charger/owners ; shadow dealers + Zechoviki - illegal factories (though if this falls through than that had the death sentence)</li>
<li>elements of civil society start to appear ;</li>
<li>dead/ossified ideology remains as glue/framework + threat of repressions; Solzhenitsyn calls for break : <a href="http://www.orthodoxytoday.org/articles/SolhenitsynLies.php">Live not by lies</a></li>
</ul>
<h5 id="perestroika">Perestroika</h5>
<p>Economy dominated by military industrial complex; agriculture inherently inefficient - the technocratic fix was to pour mountains of fertilizers, gigantic ameleoration projects, heaps of broken tractors. All this inflated industrial output by means of building equivalent of pyramids. By 1985 it was clear that they were loosing the cold war; this made apparent a lag behind the west/need for reforms;
Ossified structures made priority out of maintaining the inoperable status quo - quick succession of old general secretaries made everyone aware of the need for reform.</p>
<p>two real choices existed</p>
<ul>
<li>Russian chauvinism with less communist rhetoric; problems: Military industrial complex already had the country, also this policy was associated with ‘tightening of screws’ - that was against the interests of the ruling class.</li>
<li>Socialism with a human face / reform communism; unclear hodgepodge of ideas every class picked what it saw fit ;</li>
</ul>
<!-- -->
<ul>
<li>First stage 1985 - 1987 - slogan: ‘need to overcome isolated shortcomings’, Anti-alcohol and other campaigns.
Power remains firmly in hands of bureaucracy.</li>
<li>Second stage 1887 - 1989 - slogan: ‘Socialism with a human face’ (whatever that may be), de-Stalinization campaign, more press freedom, start of cooperatives
Main task as seen by rulers: Change of facade, legitimization/perfection of the bureaucratic barter market;
for this to happen they needed mobilization of masses by means of anti-nomenklatura slogans - the irony of Revolutions.
(Me: I think democratization is the mobilization trick that Russian elites use as temporary measure when the old order is bankrupt)</li>
<li>Third stage 1990 - 1991 - slogan: ‘transition to free market capitalism’
Administrative controls start to crumble, lead to lack of accountability on the part of bureaucrat/managers vs Moscow Ministries + no party discipline.
Ideally suited for rapid privatization, as during NEP: managers make deals with cooperatives to privatize state assets; manager uses big potential of the state to fill his pocket. Power remains in hand of bureaucracy while accountability decreases - ideal setting for ‘secret privatization’.
all big capital funds of Russia start here.</li>
<li>If the main task of Perestroika was privatization of state property into managers hand than that was a smashing success!
He says this was not planned (as in conspiracy) but that this turn of event evolved gradually, as rulers acted out basic class instincts;
all successive steps were leading to lessening of controls, thus creating opportunities for silent privatization/looting of assets.</li>
</ul>
<h5 id="russian-federation">Russian federation</h5>
<p>Author says that realistic politics means of privatization are as follows: Nomenklatura exchanges power for property; says that would be a good deal for Russia, if it leads to effective free market mechanisms that are independent from the state.
Nomenklatura would like to get property and keep power; see ideal state as liberal for insiders and restrictive for outsiders.</p>
<p>(Me: maybe this outcome was determined by the limited mandate given to reformers during 1991-1992)</p>
<p>He says liberalization of prices was the possible step on condition of disintegrating state order: real money removed the cover of pseudo state ownership;</p>
<p>challenges of Russian privatization</p>
<ol>
<li>very big scope</li>
<li>weakness of legal private business (Me: as a result the shadow economy is only source of capital)</li>
<li>lack of foreign capital</li>
<li>No former owners who claim property (me: as always the most important thing comes last)</li>
</ol>
<p>As a result the following measures:</p>
<ol>
<li>no individual approach - use of general procedures/regulations</li>
<li>did not try to combine privatization with recapitalization</li>
<li>everybody gets a small voucher check</li>
</ol>
<p>Main result/target was to stop the privatization of insiders - stop the indiscriminate looting of the manager elite; from now on they had to obey some semblance of order, come out into the open. Economic reality was introduced - so they seized building pyramids (Me: and he dares to says that Russia did not have a real shock therapy!)</p>
The CPU of the Voyager probe
2013-12-13T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/12/13/post-1
["", "blogg"] | <h2 id="the-cpu-of-the-voyager-probe">The CPU of the Voyager probe</h2>
<p class="publish_date">
30 December 2013
</p>
<p>Interesting <a href="https://news.ycombinator.com/item?id=6871607">discussion</a> on the reliability of CPU chips; stuff breaks, including CPU chips;</p>
<p>Then someone asked about how it could be that the <a href="http://en.wikipedia.org/wiki/Voyager_program">Voyager</a> spacecraft could operate for some thirty years within the <a href="http://en.wikipedia.org/wiki/Radiation_hardening">harsh conditions</a> of space; this got me <a href="https://news.ycombinator.com/item?id=6872635">interested</a> ; interesting tech indeed</p>
<hr />
<p>Very interesting <a href="http://history.nasa.gov/computers/Ch6-2.html">article</a> on the computer system of the Voyager. It turns out most of the systems is not powered for most of the time, even the component that is doing the health checks - its called CCS.</p>
<p>“The frequency of the heartbeat, roughly 30 times per minute, caused concern [176] that the CCS would be worn out processing it. Mission Operations estimated that the CCS would have to be active 3% to 4% of the time, whereas the Viking Orbiter computer had trouble if it was more than 0.2% active15. As it turns out, this worry was unwarranted.”</p>
<p>They are using <a href="http://en.wikipedia.org/wiki/Direct_memory_access">DMA</a> a lot; instruments write to memory, occasionally the CPU is turned on and picks up the new values. Also they had to manage with the fact that memory is degrading, so the system needs to adapt to working with less memory. The bus is 16 bits wide, but actually they are processing 4 bits at a time, so addition takes 4 cycles. CPU registers are stored in RAM, so probably they can reassign them if a memory cell fails.
Parts of the system were reused from the Viking mission. Also they where reprogramming the system in flight during the eighties ! That’s the reason why they could start the the mission, even without having the full software on board, the mission was extended thanks to reprogramming. Just for the Jupiter visit they had 18 software updates, think about that next time that a software update breaks something on your system.
Also its all a distributed system with several CPU’s, and some elements of redundancy, awesome tech. I guess one day alien hackers will have fun with reverse engineering this system.</p>
Review of 'Surfaces and essences'
2013-10-15T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/10/15/post-1
["", "blogg"] | <h2 id="review-of-surfaces-and-essences">Review of ‘Surfaces and essences’</h2>
<p class="publish_date">
15 October 2013
</p>
<p>Finished reading <a href="http://www.amazon.com/Surfaces-Essences-Analogy-Fuel-Thinking/dp/0465018475">Surfaces and essences / Analogy as the fuel and fire of thinking</a> By Douglas Hofstadter & Emmanuel Sander</p>
<p>I am a fan of <a href="http://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach">Goedel, Escher, Bach</a> So I was following the Author’s previous works on analogy (<a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.307.5740">here</a> and <a href="http://www.youtube.com/watch?v=n8m7lFQ3njk">here on tv</a>), and was anticipating this book eagerly.</p>
<p>This one is a very interesting book; some of the findings presented in the book:</p>
<ul>
<li>
<p>Categories are <em>not</em> clear cut ideal boxes that describe entities clearly, usually there is no clear procedure for deciding if an entity belongs to a category; indeed each category is made up of certain core concepts (pigeons and eagles are birds) and less certain fringes (Penguins are birds ?)</p>
</li>
<li>
<p>Analogy making has a dual role in creating new concepts, and in assigning entities to existing concepts.</p>
</li>
<li>creation of concepts/categories: by making repeated analogies between entities we create categories as sets of precedent cases.</li>
<li>
<p>assignment of an entity to a category is by means of making an analogy between entity and category.</p>
</li>
<li>this mechanism is in effect for multiple forms; analogies between things, and analogies between simple and complex ideas;</li>
</ul>
<p>I think there are similarities between this study of analogy and software development practices: Object Oriented design tries to divide a problem space into clear categories called classes. Now the problem is that there are many possible classifications of any problem domain; one is supposed to pick the most relevant classification that would help at solving a given problem. In practice this does not always work that way, requirements keep changing, an analysis of previous relevance can turn into a problem later on; I find it interesting that there is another form of looking at problems in the field of programming : prototype based objects, here instances of an object are copied from a prototype object, and may be altered later on; now this is similar to the idea of categories as explained in this book.</p>
<p>Intelligence is defined as the ability to quickly analyze a given situation and to abstract the essence out of it (by means of finding a proper analogy to a wider category) and to apply the gained insight efficiently; insight comes from examining the wider properties of the matched category;</p>
<p>Any problems? In the last sentence the keyword is ‘proper’; I think that this concept of intelligence needs some magic fairy dust too: I understand that the search for an analogy is equivalent to a query for a common trait/attribute or an opposite trait attribute. Now what trait is the relevant one? The same applies when inferring back from the matching category to the initial situation: is there a procedure for deciding on what is relevant to a given situation?</p>
<p>On the other hand Hofstadter is the only scientists who really tries to understand what intelligence is all about; all other specialists in AI are content with parlor trick that simulate ‘intelligent behaviour’. <a href="http://www.theatlantic.com/magazine/archive/2013/11/the-man-who-would-teach-machines-to-think/309529/">Here</a> is an article that explains the different schools of thought.</p>
<h3 id="summary">Summary</h3>
<p>If one is interested in language and thinking/cognition then this book is of much interest.
If you expect a sequel of “Goedel, Escher, Back” then you will be a bit disappointed; though a similar effect of mind expansion is achieved when reading the first two and the last two chapters.</p>
<p>I found it a bit difficult to read the book; the book follows different types of analogy, many things are therefore similar; I would often put it down and pick it up later, but then it was hard to remember what it was all about. So I made up some notes in the process of reading it; I put them up here, so as not to loose them (and to remember what the book was about).</p>
<hr />
<p>Notes on “Surfaces and essences”</p>
<p>Chapter 1.</p>
<p>Dictionary entries for words do not describe the word (any word) fully; cannot fully describe all variations of each particular definition.</p>
<p>(1) Meeting <em>in</em> five minutes and the garden
(2) Meeting <em>in</em> the garden
(3) Meeting <em>in</em> five minutes</p>
<p>Zeugmas: <em>in</em> appears in a sentence and two meanings of the same word are used in sentence (1).</p>
<p>The same sentence in different languages:</p>
<ul>
<li>
<p>Im going to brush my teeth and my hair - (the author says that ‘brush teeth’ and ‘brush hair’ are very different)</p>
</li>
<li>Voglio lavarmi la faccia e i denti. ( I want to wash my face and my teeth. - you can say that in Italian, no problemo)</li>
<li>se laver les dents (to wash ones teeth) (this is acceptable french, but more often they say: se brosser les dents (to brush ones
teeth).</li>
</ul>
<p>So these Zeugmas are very much relative to the language of the speaker (Hello Linguistic relativity)</p>
<p>Sometimes I go to work by car, and other times on foot.
In German you have ‘Fahren’ for ‘going by car’ and ‘gehen’ for going by foot, so you can’t say the same sentence the same way in German.</p>
<p>Or
The boy and the dog were eating bread - no problem in English,
In German ‘fressen’ is by animals and ‘essen’ is by people, so you can’t say the same thing in German (the same way)</p>
<p>Many more trivia like that is presented:
‘Play’ is both playing sports and music;
In italian you have giocare (for sports) and suonare (for musical instruments)
In Chinese you have different words for playing different kinds of instrument; No general word for ‘playing’ and instrument at all!
As if one has different verbs for Playing socker, playing basketball !!!</p>
<p>‘You will enjoy this zeugma as much as a piece of chocolate or of music.’
funny play on the meaning of ‘enjoy’; he says that we can parse/assign words to concepts this in fractions of a second !!!</p>
<hr />
<p>Categories:</p>
<ul>
<li>in science the categories are usually well defined as sets; statement ‘item’ is in ‘category’ is either true or false</li>
<li>Hofstadter says that assignment to categories is a relationship that returns a probability (‘shades of gray’):
therefore the decision of assigning an entity to a category is the same thing as making an analogy between entity and set of previous analogies (that is the category) (belongs to category - is like category
(‘like’ can be more or less certain - so this introduces the ‘shades’)</li>
</ul>
<p>Explains analogy: not the proportional analogies that are employed in IQ tests:
tomato : red :: broccoli : X | anser X - green (?)
(The misunderstanding is that many people reduce analogy to such relations in IQ tests)</p>
<p>The book claims:</p>
<ul>
<li>cognition is due to categorization; categorization does not come from classification -
instead categorization comes from analogy making (which returns a different measure of success (non boolean))</li>
<li>this allows us to deal with similarities of data (classification does not help here)</li>
</ul>
<p>The term ‘inference’ as used in the book: is not meant as logical induction of inference engine (as AI).
… a currently active concept is lifted out of dormancy and brought to ones attention. … For us, inference will simply mean the fact that some new element has been activated in our mind.
(not necessarily logically related)</p>
<hr />
<p>The classics on analogy:</p>
<p>Plato and Aristotle: analogy and metaphor are important mechanism in thinking, but its use can lead astray]
Kant and Nizsche follow this tradition.</p>
<p>Positivists: Hobbes and Locke: they like clear words and scorn analogy and metaphor as bullshit. science is supposedly clear and without ambiguities.
(interestingly in their writing they invoke metaphor in order to scorn it).</p>
<p>He asks: are metaphors any use? Well we draw upon experience by making analogies with past events;
‘experience’ is based on the making of analogies with past situations.</p>
<hr />
<p>Computers and Man:</p>
<p>Humans think by categorization through analogy, that’s why a Human can /infer from past events/gain insight/adapt to situations/, a computer only follows orders.</p>
<p>To bring in evolution: In order to survive Humans draw on past experience and compare/apply it to present situations. This comparison is made my means of analogy.</p>
<hr />
<p>Single member categories. My Mother for a newborn
While playing in the park this concept grows into two member concept
“That person is taking care of her just like Mommy takes care of me. That key moment marks the birth of the concept mommy with a small m. The lowercase letter is because there are two members of this new category now”</p>
<p>“What we are proposing is that the birth of any concept takes place more or less as described above. At the outset,
there is a concrete situation with concrete components, and thus it is perceived as something unique and cleanly separable
from the rest of the world. After a while, though perhaps a day later, perhaps a year one runs into another situation that one
finds to be similar, and a link is made. From that moment onward, the mental representations of the two situations begin to be connected
up, to be blurred together, thus giving rise to a new mental structure that, although it is less specific than either of its two sources (i.e. less detailed), is not fundamentally
different from them.”</p>
<p>Then we learn that every person has a Mother, making the concept still larger;</p>
<p>Then there are even more abstract usages: The Queen bee is the Mother of all bees in the hive; a Mother doll to a child doll; Mother nature …</p>
<p>So the book says that Categories are formed by applying analogies; categorization (decision if a belongs to concept b) is done by means of analogy.
The act of applying analogies is extending simple concepts/categories into more complex concepts/categories.</p>
<p>Often funny things that children say come because in the child’s mind are more inclusive then in the mind of adults:
So tom says “I want to eat some water” because ‘eating’ for him is more general than the standard usage.
.. all this is the same mechanisms as metaphorical usage of adults.</p>
<hr />
<p>In science: Galileo saw the moons on Jupiter and made the analogy onto the planets moving around the sun;</p>
<hr />
<p>Hubs: first seen as in a wheel: a center with spikes that lead out of it.
Transportation hub: an airport with lots of routes that go out of it ( concept formed by analogy with the wheel )</p>
<hr />
<p>Philosophers thought that things divide into natural/clearly defined/unambiguous categories, and that all we have to do is classify things correctly. Philosophers like to deal with platonic/supposedly objective - categories and how they relate to each other, not concerned with the procedure that can decide if instance a is a member of category b.</p>
<p>What about birds?
Wittgenstein says: a category should have</p>
<ul>
<li>necessary conditions for membership</li>
<li>sufficient conditions for membership
Definition:
the set of membership criteria are the Intention category
Instances that are in category are extension of the category</li>
</ul>
<p>Problem:
words to describe the Intention are not very exact; there is no algorithmic procedure for deciding if something is in a category.
By making up an abstract category you will get many exceptions:
If having feathers are criteria for being a bird: is a plucked bird still a bird?</p>
<p>So Hofstadter likens categories to fleeting clouds, that doe not have clear boundaries
(M.M. Ha ha: in Object Oriented programming they once had clouds for class notation (Booch))</p>
<hr />
<p>If categories are rejected: some now use the notion of Prototypes / the complete set of exemplars that one encounters;</p>
<p>(M.M. like in programming: they have prototype classes as Objects in scripting languages)</p>
<p>Another idea are mental stimulators of experiences that activate other sections of the brain.</p>
<p>Common theme: categories have stronger and weaker members; and cats. are not clearly defined. (That was a pun!)
So: Pigeons are clearly birds (near the core of the concept), and ostriches/penguins are barely birds (at the fringes of the concept)</p>
<p>Also a lot depends on context, current context has some role to play in how things are ‘categorized’ at the current moment;
(or which categories become relevant at the current moment)</p>
<p>“When Ann had to be hospitalized on an emergency basis and a transfusion was
needed, her membership in the category blood-type A+ dominated all her other category
memberships, but in a restaurant she is above all a vegetarian, while at work she is
a lawyer, at home a mother,”</p>
<hr />
<p>Metaphors:</p>
<p>Rules for metaphors: (George Lakoff, Mark Johnson)</p>
<ul>
<li>
<p>time is characterized in terms of space (metaphor for time is space): a <em>distant</em> area, from now <em>on</em></p>
</li>
<li>
<p>space is .characterized in terms of time: the first street <em>after</em> the traffic light, <em>going</em> to see them tomorrow.</p>
</li>
<li>
<p>mood is characterized by ‘high’ and ‘low’: to be very <em>down</em></p>
</li>
<li>
<p>abstract is exemplified by familiar things: a believe <em>dictates</em> behaviour</p>
</li>
<li>complex (conflicting) situation are exemplified by conflict language: outsourcing <em>kills</em> growth</li>
<li>many metaphors do not have their own category: it’s <em>greek</em> to me; a <em>cool</em> idea</li>
</ul>
<p>‘Dead metaphors’ - often used ones that are no longer original; no longer covering any original meaning; fall into disuse.
Likens it to a city: area that once was a suburb is absorbed into the core, what was once a bold metaphor is absorbed into the core meaning of a concept.</p>
<hr />
<p>Verbs and Nouns;</p>
<p>Same word is often used both as nouns and as verbs; some verbs are more abstract than others;
to snow - snow is falling
to bridge - bridge ideas; bridge between places;</p>
<p>Much - (opposite of somewhat)
comparison: much more than
quantify: wanted it so much
expectations: didn’t give me much
degree of gratitude: thank you so much
…. and lots of other situations as well.</p>
<p>Is there a clear distinction between syntax and semantics?
Maybe recognizing syntactic structures is the same mechanism as building analogies to previous experiences?
in this case analogies to previous sets of utterances are built.</p>
<p>‘I like Paris but I like Parisians’ - does not make sense, there is no conflict that but should be denoting.</p>
<p>so but might mean: however; nonetheless; an yet; still; on the other hand; … all a kind of zigzag of discourse;
and the premise is that we learn from examples to the categories/sets that explain these utterances. analogy both forms these categories
and decides when to apply them onto a given situation.</p>
<p>even a/the ; as grammatical devices they are devoid of meaning, but there is a difference between ‘a suns third planet’ and ‘the suns third planet’</p>
<hr />
<p>Linguistic relativity:
For siblings the English language has: brother and sister
In Indonesian there is: kakak - older sibling; adik - younger sibling;</p>
<p>Is it that Indoneasian has a different notion of ‘natural order’ ? One still can say ‘older sister’ or ‘female kakak’.
He says if there is single word utterance then this is more significant then multi word utterances; Well…</p>
<p>(M.M: ? Is a frequent 2-gram instance less significant then a one word utterance ?)</p>
<p>Speculative concept spaces: n-dimensional space where axis are basic measures (color, taste, shape)
He speculates that the same concepts in different languages will cover somehow different spaces;</p>
<p>Like ‘Pattern’ in English has no single word translation into French; or French ‘normalement’ has not single word translation into English.</p>
<hr />
<p>Chapter 2: composite words and proverbs (compositie entities)</p>
<p>Compound words - are learnt as a whole, nobody thinks of the etymology of words - where the word comes from.
the components of a compound word are often not ‘heard’ as part of the compound word (understand, wardrobe )
sometimes more (gentlemen) sometimes are heard more - that’s when you don’t know how to pluralize a word (teaspoonsful - teaspoonfuls)
Using a compound word does not mean that one has to understand it as a compound.
Same as with acronyms - these compress a longer utterance into something short but recognizable (?)</p>
<p>Idioms/proverbs: “Jewish Mother” as denoting person who is overprotective and complains a lot; meaning that one does not need to be Jewish or a mother in order to be “Jewish Mother”.
“Once bitten twice shy” - no animals involved; all denote a wishy category of situations, analogy used to apply these utterances.</p>
<p>Proverbs: Meaning of “He who will steal an egg will steal an ox”
1) does it mean ‘small thefts are initial steps, large thefts will follow’
2) does it mean ‘small misdeed are initial steps, large misdeeds will follow’
3) does it mean ‘small action is initial step, large action will follow’
4) does it mean ‘one thing leads to another’</p>
<p>3) probably not, because part of the message is to be on the guard against even small misdeeds; this message is lost if you allow positive actions.
Problem with abstraction:</p>
<p>Abstraction is ‘taking away the irrelevant part in order to extract the essence’ if taken to far you get out of touch.
(M.M. like in programming ;-); the question is always: is this abstraction relevant? Does it not omit too much ?</p>
<p>Are fables also just names for sets of situations that are applied by means of analogies ? Yes! ‘Sour grapes’ : failure followed by belittling the original goal.
‘Sour Grapes’ is similar to “Cognitive Dissonance” - if there is a conflict between expectations and what is possible then psychologist tell you to ‘regain a peaceful frame of mind by distorting ones perception of a troubling situation’</p>
<p>‘Sour Grapes’ situation: in which disappointment turns a person into an intellectual opportunist that is, into someone who tries to paint a failure in rosy colors</p>
<hr />
<p>Intelligence: defined here as ability to quickly analyze a given situation and abstract the essence out of it; and to apply the gained insight efficiently.
Lots of analogy making here.</p>
<hr />
<p>Chapter 3 - scope of analogy that have no one label (made up of many words)</p>
<p>Temporary categories (without a name?) : often used as queries to formulate and solve a goal;
example Category: ‘Possible birthday presents for a seven year old’ - very specific nature; no common name applies to such situations.</p>
<p>Lots of hand waving/gesturing (there - accompanied by gesture) may be used to express these temporary categories ;-)</p>
<p>And ‘Me too’ is a situation when one wants to show /some sort of/ compassion with a speaker, by making an analogy to ones own experience.
and ‘Don’t worry /that/ happens to me all the time’ - and the speaker has again made an analogy.
Now /that/ is not clear at all; (it may be in the eye of the speaker) very very ambiguous.</p>
<p>We create context by comparing an input to our own system of experiences.</p>
<p>Finding similarities between stories: And now to the problem of encoding - we are very efficient at recalling long past experiences; how are the ‘stored’/encoded in the brain?
(The brain can’t store the video of all past scenes as is (out of disk space) - it has to extract some essence; hence a need for a format / encoding of information).</p>
<ul>
<li>
<p>In stories a ‘skeleton’ story is abstracted so that similarities are found between skeleton stories (?)</p>
</li>
<li>
<p>the skeleton stories are already made up and stored when we need to recall them.</p>
</li>
<li>
<p>there are multiple ‘skeletons’ each of different generality that one may extract from a story.
Often an emotional aspect central to the ‘skeleton’ ; emotions often make up the core of an experience.</p>
</li>
<li>Part of understanding a situation is recall of past precedents ‘that remind us of the situation’ (Roger Schank)</li>
<li>Is deep essence/category extracted when a situation is experienced for the first time, and later recognition is only a recall of that encoding/past insight ?
The book says that repeated recognition of themes based on analogies is what is creating essences/categories.</li>
</ul>
<hr />
<p>Chapter 4</p>
<p>Generalizing abstraction: A is more abstract than B if set of B included in set of A.
Each thing is at the same time a member in multiple categories: Am I a man, father, human, animal ? Depends on the relevance defined by the situation/context.</p>
<p>marking: a single word can express both narrow (category a) and broader category (category b); category a is contained in category b.
Example: coffee - the drink; having coffee - might mean drinking something or eating something in a coffee house while having a conversation;
define: Marked meaning - If word used in broader sense
define: Unmarked meaning - if word used in narrow sense.</p>
<p>grow(1) - to become larger
grow(2) - to change over time</p>
<p>‘Watch out for cars’ means ‘cars’ in the wider sense, including trucks and any moving vehicles.</p>
<p>Desks: once upon a time it was a piece of furniture; with “Windows desktop metaphor” now a desk can be defined as a ‘work space’ - now desk is more general/abstract that is.
The metaphor of the “Windows Desktop” is not included back into the concept “Desk”.</p>
<p>Of concept of waves (of water) has been extended many times in physics;</p>
<hr />
<p>Different types of metaphora:</p>
<ul>
<li>
<p>completely standardized metaphorical expression, understood by native speakers; an abstract category is clearly understood by native speakers.
Jacks house is a dump(2); Dump(1) unpleasant place,garbage dump ; Dump(2) - messy place.</p>
</li>
<li>standardized expression where the source of the abstract category is no longer known (lost in time?) - but still understood by native speakers.</li>
<li>abstract category is not standardized; listeners is supposed to construct one; No standardized usage exists, but is dependent on the context.</li>
</ul>
<hr />
<p>Chapter 5 - analogies that manipulate us.</p>
<p>analogies that force decisions : analogies to past events that convince us that situation will be repeated in the future.</p>
<p>Speech errors: can reflect subconscious mental processes</p>
<p>(M.M. Freud says the same thing on slippages in <a href="http://en.wikipedia.org/wiki/The_Interpretation_of_Dreams">The interpretation of dreams</a>)</p>
<ul>
<li>here some speech errors are interpreted as ‘revealing facets of analogy making’ –</li>
</ul>
<p>Lexical blending := mixing together of stock lexical items.
When trying to express a thought: two competing expressions (each is standing for its category) come to mind; as a result both are blended into one.</p>
<p>The clerk would just turn the other eye.
turn a blind eye | turn the other cheek | turn away</p>
<p>What gets mixed up are phrases within conceptual proximity or phonetic proximity.</p>
<p>I don’t want to dwelve upon that
dwelve := dwell | delve</p>
<p>Sometimes two thoughts get mixed up (? How is that different from previous blending ?)
“I hope my package got there in one shape”
(In one piece | in good shape)</p>
<p>Mixing up concepts that are in semantic proximity:
It is easy to mix up Scandinavian countries; though difficult to mix up Finnland and Vietnam.
Or A Goose and Turkey are easy to mix up, but you wil not mix them up with a Hummingbird.</p>
<p>Or: people mix up times; or mixing up time and space (‘look at the map’ when meaning calendar)</p>
<p>Or: people mix up opposites (big/small read/write )</p>
<p>Or: concrete analogy (visual/auditory) and abstract analogy (functional/role based) reinforce each other;</p>
<hr />
<p>‘Pointless’ analogies</p>
<p>After having attended a friends 60ieth birthday, A notices fleeting occurrences of the numeral 60 and it reminds him of the birthday.</p>
<ul>
<li>
<p>the mind constantly is looking for familiar patterns; many analogies just happen and do not server any purpose (such as problem solving)</p>
</li>
<li>
<p>words denoting Sensory perception are often used to denote emotions/ideas;
Visual perception -> understanding
touch/struck/moved -> emotion
cleanliness -> morality (Macbeth effect)
sensing - the word has double meaning of perception and cognition.</p>
</li>
</ul>
<p>these similarities lead to theory of “embodyment” - all our senses lead to perception <-> emotion / thought.
The author says we are thinking ‘through the medium of our concepts, which are built up through constant analogy making, which is in response to sensory stimulus/</p>
<p>Where is abstraction here? abstraction := recognizing and isolating what different concepts have in common.
Many metaphora are ‘embodied’ / based on sensory experience somehow; so they are based on abstraction grounded in/based on sensory experience.</p>
<ul>
<li>default assumptions: often can lead astray, situation becomes difficult to solve, because default assumptions are not usually questioned
Example: Tower of Hanoi puzzle; Children tend to solve this in 27 steps, that’s because when moving from A to C the child assumes that one has to transit via B.
Abstracting ‘Moving’ as ‘State change’ (without transiting intermediate steps) is too complicated at this stage.</li>
</ul>
<hr />
<p>Analogies that exert influence:</p>
<ul>
<li>
<p>analogies to 9/11; comparing events to 9/11 creates a category ‘9/11 events’;
by comparing an event to ‘9/11’, it is assigned to a category and the category paints back on the event.
(small crash of a plane on 9/11/2004 made the stock market crash)
(Or by comparing x to the Nazis, the assigned category paints back on X, since the Nazis where evil in many ways, x is now perceived as more evil)</p>
</li>
<li>
<p>If you recently have seen X at your friends home, then you are more likely to notice X in any other settings.
If one has experienced X then one is more likely to notice X in everyday situations, one is likely to search for X and one will find it)
This is a ‘filter’/bias of sorts, it’s a good thing because it helps us to focus on recent things/relevant to us things.</p>
</li>
</ul>
<p>(M.M. like in OSX menu, the items near the mouse are displayed bigger then what is further away)</p>
<ul>
<li>
<p>Obsessions: if one is obsessed by an event, then one will see analogies to this event in almost everything;
this forced search for analogies can be a source of new insights “Chance favors the prepared mind”
(M.M. and a source of suffering, depends on the setting)
People obsessed by a hobby will often explain the world in terms of the hobby; even Pac Man can inspire a Philosophy of life.
(M.M. Philosophical Relativists/Logical positivists say that areas of knowledge are isomorphic, it isn’t but one can find lots of isomorphisms, if one tries hard)</p>
</li>
<li>
<p>‘Induction’ - extrapolating from past experience is done with more confidence if there were lots of sample points.</p>
</li>
</ul>
<hr />
<p>Chapter 6 - how we manipulate analogies</p>
<ul>
<li>Puns often work by creating ‘caricature analogies’: “Fighting for peace is like fucking for virginity”
the analogy has to a bit remote from the original situation, otherwise it does not work(not funny)</li>
<li>
<p>such ‘caricature analogies’ can explain things by puting them into different perspective, this is done by reducing an experience ot its core/conceptual skeleton.</p>
</li>
<li>is it possible to reason without analogies?</li>
</ul>
<p>Examining the question how the US got into Vietnam; transcripts of meetings show that they indeed used the analogies with historical precedents in internal debates, not just in order to explain the decision to the people.
The main arguments for the war where, ‘containment of communism’, ‘domino theory’; both where based on analogy with appeasement/Munich/Chamberlain, this in turn reinforced the argument for war.
Then argues with Mr Khong and says that one always uses analogies of some sort while reasoning…
Khong rules out analogy making because psychologists say that in order to find analogy, we need ‘surface level’ resemblance (some common non obvious trait).
These experiments are easy to construct, but lack context, so they do not prove the general case;
Also do not distinguish between domain experts and domain novices;
Each one compares as deep as his experience allows him to do; the importance assigned to trait x varies with experience in the domain.</p>
<p>Now he argues with the psychologists, so the argument gets psychological: experimenting with letter strings and similarities between them (aka the “copycat domain”)
Is abc <-> efg ?
Is abd <-> hik ?
Is aabbcc <-> abc ?
Is aabbcc <-> efg ?
Gestalt psychology says we do ‘conceptual chunking’ so aa is perceived similar to a, therefor aabbcc <-> abc
We are supposed to remember the rules how letter string is made up (encoding), as a matter of compression.
Many times choice of which analogy is more appropriate is based on matters of taste/esthetic perception.</p>
<p>Then he lost me talking about the esthetic choices between string analogies ;-)</p>
<p>Frame blends/mixing up analogies (in copycat);
How to distinguish frame blend from other categories?
In copycat: transormation aabc->aabd; what would be the similar transformation for pqrr?
pqrs? oqrr - this one blends two -> reversing the string + outmost letter and next to outmost letter are one removed.</p>
<p>Another one: this one was sitting right there (pointing) / Blending this frame (over there) with past frame of what happened.</p>
<p>Frame blends are analogies, analogies are often frame blends (lots of examples from past chapters)</p>
<p>Frame blends in translations:</p>
<p>early machine translation was like deconding a cipher: translate the word, and rearrange the result, so that syntax of source language
matches syntax of target language; Any higher level analogies are thereby lost.
Many examples show that this is not reasonable, translation is hard.
Translation is about making analogies that map concepts of the source language to that of the target language.</p>
<ul>
<li>Are we manipulating or are we manipulated by analogies ?</li>
</ul>
<p>To choose one analogy over another is to choose one viewpoint over another.
Even carefully crafted analogies come from spontaneously made up analogical links (unconscious search process).
So even the manipulator is manipulated by unconscious analogies & search for similarities.</p>
<p>Chapter 7 - naive analogies</p>
<p>Plausible but inacurate analogies with prior knowledge, made on the spot with incomplete knowledge; serve in order to make some sense of the world around;
are made spontaneously/automatic, the speaker is often unaware that he made an anlogy.</p>
<p>Naive analogies are often made by learners; so their study is important in education.</p>
<ul>
<li>Kids make analogies from what they learn into their domain; this can help them to learn/remember knowledge</li>
<li>naive analogies often coexist with the learned knowledge</li>
</ul>
<p>Lance Rips: inference of trait from typical member of category to atypical member of category is more common then
inference of trait from atypical member to typical member.</p>
<p>Does not like the notion that stuff learned in school is independent/separate from everyday knowledge / links to what we know
help to acquire stuff; Educators like to limit analogies (cause they are regarded as dangerous/leading astray), otherwise school knowledge has to build upon previously taught school knowledge.
So deduction is favored and analogy making neglected.</p>
<p>He wants to refute the notion that learned stuff is in its own realm, all recursively built up on other learned stuff: computer UI
is made up of analogies with everyday things, otherwise it would not have been adopted by a wide audience.
User interface designers try to achieve “transparency” - the illusion that the user is manipulating every-day objects;
this can only be achieved by building up analogies between the computer realm and everyday things (examples: Menus,desktop,folders,shopping carts)</p>
<p>Later, as the computer terms become familiar, they server as sources for analogies: the icon for saving stuff to disk features a diskette
(ironic because those are no longer used today)
Or: multitasking, now one hears this word to describe that a person is doing several tasks at once (on step at a time, while shifting between tasks)</p>
<p>Back to education: = sign means equivalence of left hand side and right hand side.</p>
<p>It is often understand as naive analogy: <operation> = <result> analogy; make something with the left hand in order to get the right hand side.
The <operation> = <result> analogy is often used in advertising (bye 1 = get another one for free); means that the naive analogy coexists with equivalence concept.</result></operation></result></operation></p>
<p>Another example: addition, often understood as 3 * 4 = 3 + 3 + 3 + 3 (which does not work if the numbers are not natural)
So naive analogy does not work with 3 * 0.5 ;
Also people don’t always know that a * b = b * a; even if they do they can’t explain why ;-)
Also the naive concept is that multiplication makes things bigger (what about multiplying with 0.5)</p>
<p>Situation is even worse for division; Naive concept for division is sharing n among m parties; if m < 1 then that beats the naive analogy.
Naive concept remains, if you have luck then you get a new concept that stays with the naive one, namely a/b/c/d = a/b*d/c</p>
<p>Also world problems that do 10 * 4 (done by simulating 10+10+10+10) are easier then than 4 * 10 (if one does not know about commutativity)
That is true even if you know arithmetic ! That’s because the naive way is preferred,even if the more complex one is known.
He says because in English language division implies to divide, to share; (don’t buy that)</p>
<p>(M.M. … Or pointers, lots of programmers don’t get them)</p>
<p>When solving word problem one does not immediately abstract away from it, the formal structure is not recognized right on,
so the naive concepts are also tried, if deemed appropriate those are chosen.</p>
<p>Chapter 8 - analogies that shook the world</p>
<p>we are constantly extending our categories; negative numbers are very non intuitive, so when looking for a solution to
2x+6=0; for many centuries people said that there is none;
The change came when it was found that negative numbers simplify the resolution of quadratic equations.
Even greater extension came with irrational numbers.
And n-dimensional (n>3)spaces where not quite accepted when the idea was brought up, no spatial analogy.</p>
<p>In the end things got complicated with Galois groups ;-) ever and every more abstract, the once concrete becomes an analogy to the more abstract,
then the once abstract becomes the familiar concrete, and so it goes. Physics find and application, so a very abstract concept
can turn into something very concrete.</p>
<p>(M.M. topology maps neatly into algebra; so by learning stuff in algebra one is solving by geometrical problems - algebraic topology; Now this mapping trick swas further formalized into Category theory, this is now a standard trick in modern mathematics).</p>
<p>Mechanical manipulation in math:</p>
<p>Moving variables to the second side
3x - 5 = 6x + 3
-3x = 8
x = -8/3
Two tricks; moving variable to the second side; dividing by the factor.
Recognizing this situation as being applicable means making an analogy to the ‘bag of tricks’ that one has learned.
Even more so as one goes along…</p>
<p>Physics is an area of study that values analogies; most practitioners even acknowledge its importance, and like to explain new insights in terms of everyday analogies.</p>
<ul>
<li>How Einsteins analogies changed the face of Physics</li>
</ul>
<p>Noticed similarity between formula for black body spectrum and formula of ideal gas in a container; and whee they were identical but for the following factor</p>
<p>Ideal gas = Black body spectrum
N (number of molecules) = E (total energy) / h v
So Einstein infers by analogy that there are is a light quantum; even if all of physics is sure that light is all about waves.
! And he goes for it !
So based on this Einstein finds formula for describing photoelectric effect; and o wonder, different parts of physics got unified, all this in 1905;
Nobody believed him, until 1923;</p>
<p>Now about special relativity:
Einstein looks at Galilean relativity (all centers of coordinates are equivalent) and instead of looking at mechanisms he extends
the formalism slightly: See if it holds for Mechanics + Electromagnetic (including light of course)
(Now this needs some math so he skips it).</p>
<p>E=mc^2 ; Looking at train with flashlight in same direction as movement of the train and a flashlight shining in direction opposite direction.
Einstein computes the Doppler effect based on Special relativity (Doppler effect: waves that go in same direction as train are of higher frequency, and in opposite direction are of lower frequency).
Now the thought experiment: if by emitting the light the train is loosing some energy (because of leaving photons); then a tiny amount of energy has been removed from the train;
So the total energy of the train is now a bit less; so
Kinetic energy = (mv^2) / 2 ; train moves at constant energy so mass must have changed. Wow.
So in 1905 Einstein says: M = E/c^2
In 1907 he realized that electromagnetic radiation/light has mass; that outgoing radiation takes away some mass;
In 1907 he realized that energy (in any form!) has mass; that mass and energy
are equivalent.
(this is another miraculous category extension)</p>
<p>How did he get there?</p>
<p>1905: Notion of true mass (om absolute zero) and apparent mass; the movement of atoms contribute to the total mass; difference is called ‘strange mass’
(this notion is derived from special relativity); so he thought that by emitting energy, a body is loosing some if its strange mass;
(Analogy: frozen assets in bank = strange mass; liquid assets in bank = apparent mass)
1907: Einstein does away with strange mass; any mass can potentially be converted into energy; (like in physics energy can change form)</p>
<Some stuff="" about="" General="" relativity="" that="" is="" not="" quite="" clear="">
Stanislaw Ulam: Good mathematicians see analogies between theorems or theories, but the very best ones see analogies between analogies.
(M.M. More easy to think about tiny photons leaving, if you think about their energy in terms of E=mv^2/2 then you get to E=mc^2)
Chapter 9 - Epidialog
No use to sum it up, one has to read it.
</Some>
Peak advertising
2013-10-14T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/10/14/post-1
["", "blogg"] | <h2 id="peak-advertising">Peak advertising</h2>
<p class="publish_date">
14 October 2013
</p>
<p>Peak advertising</p>
<p><a href="http://peakads.org/images/Peak_Ads.pdf">This article</a> by Tim Hwang and Adi Kamdar argues that online advertising will become less efficient over time; the argument is that</p>
<ul>
<li>younger people do not react to advertising as much as older people</li>
<li>saturation: lots of adds everywhere, so nobody gives a damn;</li>
<li>add blocking is a result of that.</li>
<li>click fraud: the opportunity makes the thieve.</li>
</ul>
<p>As a result they see the following trends</p>
<ul>
<li>advertising is concentrating around a few monopolies (also known as ‘add platforms’)</li>
<li>invasions of privacy by add platforms: add platforms want to know more about us in order to make more efficient add placements; they create a model of our interest and want to
match the adds to this model of our ‘online self’</li>
</ul>
<p>They forecast that we will see more adds as part of the online material (payed for articles as adds).</p>
<h3 id="is-that-really-so">Is that really so?</h3>
<p>It may be all true, but the competition is also bleeding: Cable TV companies seem to be suffering from falling subscriptions (<a href="http://bgr.com/2012/08/02/cable-tv-subscriber-stats-q2-2012-satellite/">here</a> )
So the audience for online content will be growing, as a result of the problems that traditional media is facing.</p>
<p>What seems to be true that</p>
<ul>
<li>lately Google is pushing much more adds in youtube.</li>
<li>Search results are very different when the Google search engine detects an intention to buy something; in this case there are much more adds; Compare search for <a href="http://www.google.com/search?q=smartphone">smartphone</a> and <a href="http://www.google.com/search?q=bye+smartphone">buy smartphone</a> - the second search is heavier on adds.</li>
<li>So far everything is fine for Google; the share price <a href="http://www.bbc.co.uk/news/business-24585998">is up</a> ;</li>
</ul>
<p>Web adds might not be very effective, but it may well take a lot of time until the advertisers have realized that. The perceived value may outlast the real value of the tool.</p>
<h3 id="is-there-another-way">Is there another way?</h3>
<p>Is there a better way to advertise, one that does not require these companies to gather all scraps of data that it can get hold off.
Is it really that important to spy after the users?</p>
<ul>
<li>There is the possibility of merging online content and advertising:
Look at Amazon, while looking at book reviews, one also gets an add for related books “People who bought this book also bought …”
Amazon is making a lot of money around that offer.</li>
<li>It may be enough to understand the online content, and to pair in adds that are related to online content.</li>
<li>A better understanding of the served content vs served adds would require less intrusive user data gathering on the part of the ‘add platforms’</li>
<li>Also there seems to be at least some degree of consumer backslash: The search engine <a href="https://duckduckgo.com/">Duck duck go</a> promises more privacy than
its competitors; So thanks to the Snowden revelations it has seen some <a href="https://duckduckgo.com/traffic.html">remarkable growth</a> So far that may be a “nerdy” trend, but over times “nerdy” trends tend to be accepted by the larger public.</li>
<li>Placement of adds based on the served online content is quite a technological challenge, this would open up opportunities for newer players.</li>
</ul>
<p>Well, I think it is hard to tell, when tech is involved you have a constantly changing picture…</p>
<h3 id="isnt-that-too-hard">Isn’t that too hard?</h3>
<p>Currently the adds are separate from the content – I think its a matter of feasibility.</p>
<ul>
<li>Placement of adds can be made on popular pages, the relevantly paired content may not get any views at all!</li>
<li>Current system of paring adds onto a constructed user profile is a task of modern practical data driven AI. Now merging of adds and content would require machines that are able to adjust to the meaning of a text; the machine would have to somehow understand what is written in the text in order to put in a relevant add; Now that’s really hard.</li>
</ul>
<p>Maybe the current ways will reach a limit (don’t know when); it seems that at some stage modern data driven AI will reach its limit; when this happens we may need real machine intelligence;</p>
<p>However the road to real machine intelligence is a long one, so far only Douglas Hofstadter is looking at that problem.
Also <a href="http://www.theatlantic.com/magazine/archive/2013/11/the-man-who-would-teach-machines-to-think/309529/">this article</a> describes the different concepts of AI well.</p>
The secret policeman's crystal Ball
2013-06-08T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2013/06/08/post-1
["", "ramblings"] | <h2 id="the-secret-policemans-crystal-ball">The secret policeman’s crystal Ball</h2>
<p class="publish_date">
08 June 2013
</p>
<p>The NSA has a huge surveillance program called <a href="http://en.wikipedia.org/wiki/PRISM_(surveillance_program)">PRISM</a> ; now a key point of the program is a large <a href="http://en.wikipedia.org/wiki/NSA_call_database">call data base</a> (story was <a href="http://www.guardian.co.uk/world/2013/jun/06/nsa-phone-records-verizon-court-order">first reported here</a> ); Here is what Mr. Schneier has to <a href="http://www.theatlantic.com/politics/archive/2013/06/what-we-dont-know-about-spying-on-citizens-scarier-than-what-we-know/276607/">say on the subject</a> .</p>
<p>This call database contains records on all calls that have been made between two telephone numbers. A record contains the following data:
* phone number of the caller and the destination of the call
* time and duration of the call.
* location data from where the call has been made.</p>
<p>If things are viewed from a data processing perspective than it is possible to understand the following; (Disclaimer: my theories might also be wrong, I am not an expert in the areas of Security or law and order)</p>
<p>The real value of a call database is that it can be used to build a <a href="http://en.wikipedia.org/wiki/Social_graph">social graph</a> . A social graph consists of nodes: each telephone number (standing for a person) is a node in the graph; two nodes are connected if there has been a phone call between two phone numbers; a connection between two nodes in the graph is called a link in the graph. It is possible to assign attributes to links: if there have been many calls then the attribute that shows the strength of link gets a higher value; timing and activity of calls can also be assigned weights, so that recent calls are of higher value, etc. etc.</p>
<p>A social graph shows exactly how individuals who make up society as a whole are interconnected. If a political movement has a leader, then the social graph can show the following</p>
<p>* what is the immediate circle of contacts around a leading personality</p>
<p>* what is the larger cluster of persons who are influenced by the leader person; this is achieved by following the immediate circle contacts and then the contacts of the contacts, etc.</p>
<p>* what is the intersection of different clusters of interest; this could mark the degree how certain movements influence one another.</p>
<p>* what are the dynamics of influence: did a leader manage to gain a wider following, what was the time frame when influence was acquired. etc. etc.</p>
<p>* Malcom Gladwell’s <a href="http://en.wikipedia.org/wiki/The_Tipping_Point">Tipping point</a> argues that social movements are usually initiated by a small number of persons, by those who tend to make a difference</p>
<blockquote>
The success of any kind of social epidemic is heavily dependent on the involvement of people with a particular and rare set of social gifts. ... the 80/20 Principle, which is the idea that in any situation roughly 80 percent of the 'work' will be done by 20 percent of the participants.
</blockquote>
<p>Clustering of the social graph can identify persons with unusually high level of connectivity / activity. These persons can then be identified and could be targeted for further influence. One can create a ‘Who is Who’ database of multiplier persons who could wield a wider influence.</p>
<ul>
<li>It is possible to apply statistics / machine learning algorithms in order to extract significant clusters from the data; this would effectively create a map of society; A dossier on every person is feasible.</li>
<li>Location data can track the movement of subscribers; it is possible to check if a given person has taken part in a protest action or not; it is possible to know if a person has attended a public meeting; Lots and lots of possibilities, and it is all data that can be used in court against a plaintif.</li>
</ul>
<p>Lots of things are possible with such data; it is of course also a question of cost vs benefit, we don’t know what measures were actually implemented in the end. We also don’t know what kind of data was gathered from the big technology companies. I know that law enforcement organizations like to extend their scope and reach, just because of the possibility and out of institutional interest. The question of oversight and accountability is crucial, but these are <a href="https://www.eff.org/deeplinks/2013/06/government-says-secret-court-opinion-law-underlying-prism-program-needs-stay">obviously lacking</a> ; the NSA anual budget is <a href="http://money.cnn.com/2013/06/07/news/economy/nsa-surveillance-cost/index.html">estimated around 10 billion $</a>, so the possibilities are huge.</p>
<p>Is this knowledge effective at fighting terrorism? I don’t know; terrorist organizations are very sectarian; it should therefore be relatively easy to figure out a wider circle of connections that originate from a close nit circle of suspects. Of course tracking a defined small circle costs a fraction of what is required to track the whole population;</p>
<p>It might be argued that sweeping surveillance measures are needed in order to monitor “sympathizers” of terrorist movements. I don’t know, not so long ago back the 1970ies, West Germany had the problem of left-wing <a href="http://en.wikipedia.org/wiki/Red_Army_Faction">RAF terror</a> , <a href="http://de.wikipedia.org/wiki/Radikalenerlass">special laws were passed</a> the state reacted by monitoring suspected radicals; some <a href="http://en.wikipedia.org/wiki/German_student_movement#The_effect">3.5 million individuals were investigated and 10,000 refused employment</a> ; meaning they were not allowed to work as civil servants . For a democracy, that’s quite a lot of papers turned, the West German state was worried about infiltration, it was worried that leftists would implement a <a href="http://en.wikipedia.org/wiki/Rudi_Dutschke">long march through the institutions</a> and gain positions of influence in the bureaucracy.</p>
<p>What is learned here is that the measures that are taken by the state are proportional to the threat level perceived by the state; I don’t know if such a strict level of surveillance is justified by the current level of insecurity, is this the real state of the union? The BBC <a href="http://www.bbc.co.uk/news/world-22804547">tries to explain the situation</a> during the Iraq war the NSA got into a habit of tracking the metadata of each and every conversation - the trick of following the social graph of terror suspects was an efficient tool during <a href="http://en.wikipedia.org/wiki/Iraq_War_troop_surge_of_2007">the surge</a> . However the threat of terror in Iraq was much much larger than what we have now; Iraq saw suicide bombings on a daily basis.</p>
<p>It is possible that the perceived risk of terrorism in the US is out of step with the real risk, it is also likely that the suggested ‘fix’ to the problem is not effective, and that it creates new problems; Bruce Schneier <a href="http://www.theatlantic.com/politics/archive/2013/06/the-irrationality-of-giving-up-this-much-liberty-to-fight-terror/276695/">says so</a></p>
<p>Is it all about policing of internal dissent, about monitoring movements like the <a href="http://en.wikipedia.org/wiki/Occupy_movement">occupy movement</a> ? Might be, but I guess that would not be too constitutional. The crash of 2008 has taught us that society is very brittle, less stable than what we got used to think.</p>
<p>Are these measure introduced, just because technology makes it feasible to snoop on us ? Maybe the element of preemption plays a factor: if the government does not log the whole traffic then competing powers might do it, so we better do it just in case. It is hard to tell if this is a real factor. Is sweeping surveillance just another tool in an ongoing cyber arms race ?</p>
<p>Lots of questions here, not too many answers are forthcoming.</p>
<h3 id="how-did-we-get-here">How did we get here?</h3>
<p>I think it is an unlucky combination of factors</p>
<ul>
<li>Each law enforcing agency is always grabbing as much power as it can get;</li>
</ul>
<!-- -->
<ul>
<li>the Patriot Act gave them the license to do expand the scope of snooping.</li>
</ul>
<!-- -->
<ul>
<li>the technology makes it possible; digital trail that everybody is leaving is very significant, so somebody will always try to grab it, be it corporations or the state. Nature does not tolerate a vacuum of power. Sen. Ron Wyden says: “It used to be that the limits on technologies were to a great extent a form of protection for the American people. A lot of that seems to be going to the wind.” - <a href="http://www.rollingstone.com/politics/news/q-a-senator-ron-wyden-on-nsa-surveillance-and-government-transparency-20130815">link</a></li>
</ul>
<!-- -->
<ul>
<li>the nanny state is much about extending command and control over its subjects. I think that Eric Schmidt is unfortunately right here: ‘surveillance … is the nature of our society’ <a href="http://www.theguardian.com/world/2013/sep/13/eric-schmidt-google-nsa-surveillance">link</a> .
Bruce Schneier formulates it like that : <a href="https://www.schneier.com/news/archives/2014/04/surveillance_is_the.html">Surveillance is the Business Model of the Internet</a> ; Corporation gather a lot of data on us, so that they will be better at placing adds that are matched to our profile; this mountain of data is in private hands and government agencies just have to tap it.</li>
</ul>
<!-- -->
<ul>
<li>power is addicting / secret power must be very addicting.</li>
</ul>
<!-- -->
<ul>
<li>The cold war has finished some time ago; so nowadays there is no need for moral superiority over an ideological enemy. Nowadays anything that strengthens the state is good for us;</li>
</ul>
<!-- -->
<ul>
<li>There is also a growing sense of insecurity, its not just the threat of terrorism, I think that even an inflated threat perception would not explain the monitoring of all communications, there are more cost effective ways to isolate and deal with violent sects; It is a general sence of insecurity; maybe its the fears of a sudden financial crisis that can bring about upheavals; maybe there are other potential threats. I guess the security apparatus is called uppon to compensate for that sense of general insecurity. ( more <a href="http://mosermichael.github.io/cstuff/all/ramblings/2011/10/27/forces-of-history.html">here</a> )</li>
</ul>
<p>I don’t know how you want to put the genie back into the bottle, part of the equation is that society would have to change here; just seems to be an impossible problem to solve.</p>
<p>An intersting observation is that social graphs are essentially instruments of surveillance / social control. Maybe some of the value that is somehow factored into the share price of facebook and google is the larger political significance of all those huge mountains of data that these companies are gathering on us. It is also interesting (although disgusting) to observer that in some way surveillance and social control of large groups of people are now the motors of technological progress, at least in the field of data processing.</p>
<p>Also I guess the NSA should adopt this Stasi anthem; it says something like ‘Tell me where you stand and which path you are on’</p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/tyShCNbsplI" frameborder="0" allowfullscreen=""></iframe>
<p>The STASI says:</p>
<p>“Wir haben ein Recht darauf, dich zu erkennen.
Auch nickende Masken nuetzen uns nichts.
Ich will beim richtigen Namen dich nennen
und darum zeig mir dein wahres Gesicht.”</p>
<p>My translation:</p>
<p>(following some incantions that hide imanent conflicts that originate from supressed (prussian) upbringing, all that combined with Soviet paranoia (that’s what you had in the GDR))</p>
<p>“we have the right to recognize you/nodding masks (disguises) are of no use to us/I want to name you by your true name/therefore show me your true face” - apparently speaking here for the ministry of state security.</p>
<p>.</p>
<p>Wolf Biermann made the following poignant observation: the realities of East German socialism was that of computer Stalinism, where dissenters receive a special hole in the punch card of their personal file, whereas Stalin would punish dissenters with a bullet hole in the neck ( <a href="http://www.spiegel.de/spiegel/print/d-41843129.html">here</a> ) . In a way he saw this a process of gradual liberalisation of the stalinist system, however the power structures of the Stalinist system would remain unchanged. So much for computers being the technology of liberation, these are just tools that can be used for any purpose …</p>
<hr />
<p>In the Sci-Fi novel <a href="http://en.wikipedia.org/wiki/The_Caves_of_Steel">The Caves of steal</a> by Isaac Asimov there is an overpopulated and heavily urbanized planet Earth vs the space colonies; The space colonies are rich and have a low population density, unlike Earth they tolerate and use Robots. The plot of the novel is a detective story, where detective Elijah Baley is working together with Robot R. Daneel.</p>
<blockquote>
R. Daneel came to his desk with a sheaf of papers.
"And those are?" asked Baley.
"A list of men and women who might belong to a Medievalist organization."
"How many does the list include?"
"Over a million," said R. Daneel. "These are just part of them."
"Do you expect to check them all, Daneel?"
"Obviously that would be impractical, Elijah."
....
Baley said, abruptly, "How did you get your list?"
"It was a machine that did it for me. Apparently, one sets it for a particular type of offense and it does the rest. I let it scan all disorderly conduct cases involving robots over the past twenty-five years. Another machine scanned all City newspapers over an equal period for the names of those involved in unfavorable statements concerning robots or men of the Outer Worlds. It is amazing what can be done in three hours. The machine even eliminated the names of non-survivors from the lists."
"You are amazed? Surely you've got computers on the Outer Worlds?"
"Of many sorts, certainly. Very advanced ones. Still, none are as massive and complex as the ones here. You must remember, of course, that even the largest Outer World scarcely has the population of one of your Cities and extreme complexity is not necessary."
</blockquote>
<p>Here it is, a book from the 50ies explains that one needs complex computers in order to cope with dissent in over complex societies. Good Sci-Fi is better than a crystal ball;</p>
<hr />
<p>Also the movie <a href="https://mdb.com/title/tt0181689">Minority report (2002)</a> is about this certain lack of privacy; here is what wikipedia has to say <a href="http://en.wikipedia.org/wiki/Minority_Report_(film)#Media">here</a></p>
<blockquote>
"Spielberg conceived of the idea of a future world permeated with intrusive capitalism and government surveillance after everyone at the "think tank summit" told him that "the right of privacy is a diminishing commodity" which will soon be thrown "right out the window".\[30\] According to film critic J. Hoberman, Minority Report "visualizes (as well as demonstrates) a future where the unconscious has been thoroughly colonized."\[96\] When the movie first appeared in theaters a common source of reviewers' complaints was the film's product placement, which they found intrusive.\[97\] The personalized advertising is disconcerting partly because of the invasion of privacy, but also, argues Cooper, because it is cold, impersonalized, and insincere.\[93\] Film scholar Martin Hall says that the purpose of the ads Anderton runs into are "encouraging him to buy certain products and, by extension, affirm his place in society."\[98\]
</blockquote>
<p>Spielberg said in a press conference in 2002: <a href="http://www.movingpictureshow.com/dialogues/mpsSpielbergCruise.html">here</a></p>
<blockquote>
"I want to be clear about this: I'm not an advocate of pulling back the CIA's and the FBI's far-reaching powers right now," Spielberg said. "I think this is a time of war, and they need to do what Lincoln did when he suspended the writ of habeas corpus in 1862… During times of war, things like that have to happen.
What I'm worried about is when we have finally gone beyond the brink, where we are right now, and things start to settle down. Will the government pull back those powers of surveillance? Or are they going to say that's the new standard for them? Like, 'Hey, you've lived with them for five years. Sorry, folks, but that's just the way it's going to be from now on.' I hope that doesn't happen. That would be very sad. If this doesn't end, then we'll have to go back to the college campuses and hold up signs."
</blockquote>
<p>Interesting that i always seem to ‘get’ things with a ten year delay. I guess the message of the movie was a bit veiled on purpose, so that people would not object too much, given the terrible terror attacks of 9/11.</p>
Linguistic Relativity and Me
2013-06-02T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2013/06/02/post-1
["", "ramblings"] | <h2 id="linguistic-relativity-and-me">Linguistic Relativity and Me</h2>
<p class="publish_date">
02 June 2013
</p>
<p>A <a href="http://www.edge.org/conversation/how-does-our-language-shape-the-way-we-think">very interesting article</a> by Lera Boroditsky on how language shapes the way we think.</p>
<p>The Pormpuraaw people in Australia express spatial relations in terms of east, west, north, south instead of right, left, up, down; They have to stay oriented all time, otherwise they have no way to express themselves. Incidentally they are also great experts at navigating in unknown terrain; simply no match to most of us in this regard.</p>
<p>The quoted article asks if this a property of culture or a trait of the language? I gather from the article that it works both ways, although that the influence is limited;</p>
<p>Here are my personal observations: I am a native speaker of German and my job is that of a computer programmer. I think that the German language trains me at understanding <a href="http://en.wikipedia.org/wiki/German_sentence_structure">long wound sentences</a> ; and that such a skill is of benefit to my work.</p>
<p>I often have to fix errors and adapt an existing program to changing requirements. This means that I have to read and understand existing programs more often than I have to write new ones. I think that the process of reading a program is similar to the process of parsing a very complicated sentence.</p>
<p>It is sometimes not easy to understand sentences with a complex structure of subordinate phrases. The first task is to listen patiently and to analyze how the sentence unfolds; keep listen patiently until one hears a closing Verb; next task is to work in the reverse direction from the end of the sentence back to the start. Only during this reverse movement one can reconstruct the meaning of what has been said.</p>
<h3 id="reading-programs">Reading programs:</h3>
<p>The very first step is to understand the new system in terms of its input and its output; If it communicates with the network then what are the details of the communication protocol that is used? If it is a program that interacts with the users, then what entities are manipulated by the user? A system interacts with its environment in many ways; what are the main, most often used scenaria that characterise the system?</p>
<p>The next step is to understand the big picture: this means the breakup of the system into modules and sub systems; a module contains source text that it intended to implement related functionality within a program. For this stage one needs to apply some knowledge of software engineering and lots of practical experience in the field. During this stage we search for common metaphors and idioms: how is memory managed? How are events logged within the system? How are networking events handled ? What paradigms of multiprocessing/multi-threading are being used?</p>
<p>Once this is understood, search for the main entry point: How does the request, the external stimulus enter the system ? Also a program has to manipulate data: what are the data structures that represent the request within the system ?</p>
<p>Once all this is understood, read the code and mentally work through the sequence of steps that are executed as the request is being processed; All this continues our attempt to understand the main use cases / the main scenaria that need to be understood. While doing so search for additional clues; It sometimes helps to look at both the source code and at the logs that were produced by the system. At this stage we are actually reading the code of functions and objects that make up the program. A function often contains many nested conditional statements (if a then if b then if c else else else). Here we have our complex phrase structure; again we have to wait for the many partial statements, until there is a Verb; the action here most often is a function call that is supposed to implement a different aspect of the system.</p>
<p>Often one has to work backwards: a new detail that is discovered adds clues that clarify the previous layers. It also may happen that one needs to return to a previous stage in this process, new data can often revise prior assumptions.</p>
<p>It is here, I think, that a habit of backtracking, that has been trained by the habit of parsing long wound sentences, is of rather great benefit. (The last sentence is also supposed to be a pun on the whole argument ;-)</p>
<p>On the other hand it is impossible to know if all this was all that important. What was more important: some training in the parsing of long wound sentences or certain other non-functional aspects of my career? I have been working mostly for well established companies, these companies have a priority to maintain existing systems. Programmers who work for start-ups need to write new code more often than me.</p>
<h3 id="more-on-long-wound-sentences">More on Long wound sentences</h3>
<p>The <a href="http://de.wikipedia.org/wiki/Nebensatz">German Wikipedia entry</a> says that the practice of using a complex word structures has been in lesser use since the 20ieth century; funny that this observation so far did not make it into the <a href="http://en.wikipedia.org/wiki/German_sentence_structure">English Wikipedia article</a>, it might be that outside observers find it harder to notice such a tendency for simplification.</p>
<p>I can only speculate as to why this habit of expression has been flourishing in the first place.</p>
<p>Maybe the Buerger of the 19th century where eager to ascertain themselves as being better than the aristocracy; They made a point of standing on higher ground than those who held the real political power throughout the 19th century. One of the ways to do so was <a href="http://en.wikipedia.org/wiki/Bildungsb%C3%BCrgertum">Education/Bildung</a> . It might be that complex language was a tool in the political struggle of the day. <a href="http://en.wikipedia.org/wiki/Thomas_Mann">Thomas Mann</a> , the writer of the Bildungsbuerger incidentally made a lot of use of this style. On the other hand this style is much older, some of the <a href="http://en.wikipedia.org/wiki/Critique_of_Pure_Reason">most convoluted sentences ever</a> were written in the 18th century by Kant, so go figure.</p>
<p>Does it also show a certain pretentiousness of the speaker? Does it mark the desire of the speaker to stress his personal distinction by making his own knowledge less accessible to others?</p>
<p>Or is this a clever ruse of packaging; is it a trick that forces the listener to invest more into the parsing of a sentence; maybe the plan is to create additional involvement, to trick the listener into investing more effort at understand the message of what has been said?</p>
<p>What is real reason here? I haven’t the foggiest idea.</p>
<p>The choice of style still remains the individual choice of an author, for that matter one can choose to talk in simple ways, or one can talk like the character <a href="http://en.wikipedia.org/wiki/Owl_(Winnie_the_Pooh)">Owl in Winnie the Pooh</a>.</p>
<h3 id="how-much-does-it-all-matter">How much does it all matter?</h3>
<p>I can understand that language creates habits of expression; it also forms common idioms that are understood by a wider audience. Works of art are often based on allusions that have to be understood by a wider audience. Probably all public communications is making assumptions as to what can be understood by the intended audience.</p>
<p>But I wonder: does this common system of associations predetermine the thoughts and mental associations of an individual ? Do the conventions of the common language place <a href="http://en.wikipedia.org/wiki/Sapir%E2%80%93Whorf_hypothesis">limits on what can be thought</a> ?</p>
<p>I would argue that besides the public language many individuals are forming their own private language of idioms over the course of their lifetime. A system of mental associations is a language - any system of signs is a language. This private customized language serves us well, is it also sometimes shared among a close circle of associates.</p>
<p>As a student of mathematics I noticed that the material of a course was only understood when I could create at least some basic analogies between mathematical concepts and my realm of experience. Now I think that these analogies together were forming my own private language, this language was extending the limits of thought for me. (Well, I wasn’t a big success as a student mathematics, but still…)</p>
<p>I would argue that the ability to create a private system/language of associations creates a huge degree of freedom for an individual. I think that on an individual level the Saphir Worf conjecture is certainly false.</p>
<p>Also from the quoted article:</p>
<blockquote>
How do we know that it is language itself that creates these differences in thought and not some other aspect of their respective cultures?
One way to answer this question is to teach people new ways of talking and see if that changes the way they think. In our lab, we've taught English speakers different ways of talking about time. In one such study, English speakers were taught to use size metaphors (as in Greek) to describe duration (e.g., a movie is larger than a sneeze), or vertical metaphors (as in Mandarin) to describe event order. Once the English speakers had learned to talk about time in these new ways, their cognitive performance began to resemble that of Greek or Mandarin speakers. This suggests that patterns in a language can indeed play a causal role in constructing how we think
</blockquote>
<p>For the purpose of the test people did adjust their own personal language to the test requirements;</p>
<p>Since of the private nature of the personal language of associations, it is an impossible subject of study. However there are many questions that are impossible to study; a similar such question is ‘to what extend does the placebo effect help on an individual level’. Maybe its also good that this subject cannot be studied. Those who are looking at a person from a position of power will never have this tool of knowledge at their command. The benefit of doubt is a good thing.</p>
<h3 id="the-sinister-politics-of-linguistic-relativity">The sinister politics of Linguistic relativity</h3>
<p>Somehow the concept of Linguistic relativity has had some strong political implications:</p>
<p><a href="http://en.wikipedia.org/wiki/Sapir%E2%80%93Whorf_hypothesis#German_Romantic_philosophers">Source here</a></p>
<blockquote>
Von Humboldt argued that languages with an inflectional morphological type, such as German, English and the other Indo-European languages were the most perfect languages and that accordingly this explained the dominance of their speakers over the speakers of less perfect languages.
</blockquote>
<p>Here it is: all languages are equal, but some languages are more equal than others; and this notion is invented exactly during an age of colonial expansion. Each age gets the thought that it needs.</p>
<p>The notion that thought is shaped by language is also a bit totalitarian, as it does not allow for alternatives. Welcome to the language of totalitarian propaganda; Both the Nazis and Soviet power regarded their respective language of propaganda as a source of power.
The Nazi language of propaganda has been analyzed by Victor Klemperer in <a href="http://en.wikipedia.org/wiki/LTI_%E2%80%93_Lingua_Tertii_Imperii">LTI Lingua Tertii Imperii: Notizbuch eines Philologen</a> - also see this <a href="http://lti-lexikon.de/">lexicon of the Nazi propaganda language</a> based on this work. I think that the Soviet language of propaganda has been an inspiration to Orwell’s <a href="http://en.wikipedia.org/wiki/Newspeak">Newspeak</a></p>
<p>Heidegger is often quoted in the context of Linguistic Relativity, well the man was a <a href="http://en.wikipedia.org/wiki/Heidegger_and_Nazism">prominent Nazi</a> and might have had some influence on the formation of the <a href="http://lti-lexikon.de/">Lingua Tertii Imperii</a> .</p>
<p>I would also like quote an <a href="http://unishade.com/gimle/index.cgi/insomanyotherways.comments">interesting opinion</a> by Baldur Bjarnason</p>
<blockquote>
Linguistic relativism is the equivalent of staring down the barrel of a gun while ignore the person whose finger is on the trigger.
Language, linguistic dominance, are the cannons of cultural warfare. Without a language, a culture is defenseless.
The linguistic relativists might be right in all of their observations, but they are simply staring at the bullet and mistaking it for the lock, stock, barrel and sniper all rolled into one convenient lump of lead.
Language is wielded, formedyour arms and armour.
It kills. Just ask the Welsh, Kenyans, Native Americans or South-American Natives.
Linguistic relativism is a nice idea to those who belong to a dominant, still imperialistic culture (and this applies to the English, Japanese, Koreans and Germans, all cultures that are strong and on the offensive in the war of globalisation).
But there is nothing relative about a bullet in the head.
Or fighting for the survival of your nation and culture.
The odds are stacked against us, in your favour.
</blockquote>
<p>Therefore beware, beware of the Linguists;</p>
<p>Or rather beware of the philosophers of language that is.</p>
First post on the github blog
2013-05-19T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/05/19/post-1
["", "blogg"] | <h2 id="first-post-on-the-github-blog">First post on the github blog</h2>
<p class="publish_date">
19 May 2013
</p>
<p>I am migrating away from google+; my <a href="http://mosermichael.github.io/cstuff/blog.html">stuff is now here</a></p>
<p>I have a script that helps in migrating a google+ based blog to github blog that is based on textile:</p>
<p><a href="https://github.com/MoserMichael/cstuff/blob/master/scripts/google%2B-migration.pl">the script</a></p>
<p>I did not like the latest google+ redesign; also there are some posts that the google masters classified as ‘in violation of their terms of use’ whatever that means; as for me that violated my terms of use, so now I am moving.</p>
<p>Also this social network thing is increasingly about control of the customers information diet / information bubble. This functionality includes an element of filtering: the system does not want you to be overwhelmed by/spammed with frequent posts, so it filters the information and shows you a subset of the postings. How is this filter implemented? Is it keyword based, is there a bias against certain keywords? Filtering is a close relative of censorship; many aspects of those social networks are not very transparent, and I suspect that those aspects will be subject to abuse in the near future.</p>
<p>The old internet (Wordpress blogs/RSS/lack of a controller that records the social graph) is less centralized, and better for freedom.</p>
<p>Maybe its all part of a wider pattern; Once upon a time computing was centralized, and the mainframe ruled. Then came the PC and everybody would have a computer in his home, people would invent new applications for the computer, and these in turn would enable people with more knowledge and means to exchange information; Nowadays computing is moving back to the cloud; the cloud resembles the the mainframe in many aspects; The mainframe invented virtualization with VM/CMS,VM/ESA; now the cloud is using virtual machines to isolate processing within the cloud. Half of the mainframe is about fault tolerance and high availability, so is the cloud. Now with this higher centralization of computing we are increasingly dependent on the benevolence of the rulers;</p>
<p>But things may change again; hardware changes, software changes too. People forget the old ways and new ones are created. Nothing is fixed.</p>
A song...
2013-05-13T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/05/13/post-4
["", "blogg"] | <notextile>A song...</notextile>
<hr />
<p class="publish_date">
13 May 2013
</p>
<p>This ever changing post is about Music</p>
<hr />
<p>For me this is the ultimate Rock & Roll song of my generation;
It’s very much about the end of Soviet / post soviet periods.</p>
<p><a href="http://www.youtube.com/watch?v=P1nAnLzcfrY">Аквариум Этот поезд в огне</a></p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/P1nAnLzcfrY" frameborder="0" allowfullscreen="">
</iframe>
<p>Still, for me the essence of Rock & Roll music is deep introspective but nevertheless nihilistic bitching; For me the crisis of Rock & Roll goes hand in hand with this approach: it gives relieve to the ailing soul, but it does not give answers. I can see this clearly in the <a href="http://www.youtube.com/watch?v=_THqHI5y2KI">interview with Boris Grebenchekov</a></p>
<p>Here is my translation of the lyrics:</p>
<p>This Train’s in flames by <a href="http://en.wikipedia.org/wiki/Boris_Grebenshchikov">Boris Grebenshikov</a></p>
<p>Colonel Vasin got back to the frontline
He took along his young wife
Colonel Vasin gathered his troops
and declared ‘let’s get back to our homes
We’re already fighting for seventy years
And they taught us that life is a battle
It’s the Intel that established as fact
Those who we fought were our own selves’</p>
<p>( or rather: Those who we fought were of our own’ )</p>
<p>Saw generals in high places
They eat and drink our death
Their kids all go crazy
Cause there’s nothing for them left to want
And the land is left corroded
And churches are mixed in with dust
If we want that there’s some spot to return to
Then now is the time to come home”</p>
<p>This train is in flames, and no levers are left to pull
This train is in flames, and there’s no place to run to,
This land it was ours, until we got trapped up in the fight
As no-mans land it will perish; time to return it to ourselves</p>
<p>Torches burn all around us
All demised troops are summoned up
And people who shot at our fathers
Make plans to finish off with our kids
We were born to the sound of the marches
They menaced us with gaol
But enough, enough of the grovel
We have already returned back home</p>
<p>This train is in flames, and no levers are left to pull
This train is in flames, and there’s no place to run to,
This land it was ours, until we got trapped up in the fight
As no-mans land it will perish; time to return it to ourselves</p>
<p>Other songs by the Grebenchikov</p>
<iframe width="420" height="315" src="//www.youtube.com/embed/8SbOAFrZDAE" frameborder="0" allowfullscreen="">
</iframe>
<iframe width="420" height="315" src="//www.youtube.com/embed/LcgywNSTRGE" frameborder="0" allowfullscreen="">
</iframe>
<hr />
<iframe width="560" height="315" src="http://www.youtube.com/embed/5i15CjMG0WE" frameborder="0" allowfullscreen="">
</iframe>
<p>I don’t agree with the song, except for one magical phrase:</p>
<blockquote>
"Nehmt euch die Freiheit, sonst kommt sie nie!"
<p>
"You Take yourself the freedom, else she will never come"
</blockquote>
That's it: freedom is always an initiative that fights itself up from below; i guess that has been established - ever since the Magna Carta.
and that's the magic of open source software; the freedom to use the computer as it suites you, fought for by geeks like me.
(However: most closed systems like phones work on operating systems derived from BSD or Linux; so in the end everything can be corrupted)
<blockquote>
Что один человек сделал другой завсегда сломать сможет - Кузнец из "Формулы любви"
What is made by one man can always be undone / broken by another
Character of Blacksmith from the film "The formula of love"
</blockquote>
------------------------------------------------------------------------
<iframe width="420" height="315" src="//www.youtube.com/embed/EWuotfT_j6s" frameborder="0" allowfullscreen="">
</iframe>
A very existential song; so much compressed into just a few words.
The commercial that says it all, now thats art.
<blockquote>
Boy: What's for tea, Mum?
Grown up: What's for tea, darling? (pause) Darling, I said "what's for tea?"
old Gent: What's for tea, daughter?
Heinz baked beans
</blockquote>
------------------------------------------------------------------------
Ost/West Überblick.
<blockquote>
... graue menschen mit grauen gedanken glotzen auf mich denn ich bin nicht grau. ICH BIN BUNT ICH BIN BUNT ICH LIEBE FARBEN ROT SCHWARZ GRÜN BLAU!!!
</blockquote>
<blockquote>
... gray people with gray thoughts stare at me - cause I'm not gray;
I'm colorful; I love colors Red,Black,Green,Blue!!
</blockquote>
<iframe width="420" height="315" src="http://www.youtube.com/embed/0aO-ykEv4Nw" frameborder="0" allowfullscreen="">
</iframe>
Der Goldene Reiter
<iframe width="420" height="315" src="//www.youtube.com/embed/sZhzgK5BOx4" frameborder="0" allowfullscreen="">
</iframe>
"neue Behandlungszentren bekämpfen die wirkliche Ursache nie." - "New treatment centers don't deal with the real causes"
.... and now we have the "proper East-West perspective"
"ich steig aus um wieder da zu sein" "I drop out in order to be back in"
Ideal - Berlin
<iframe width="420" height="315" src="//www.youtube.com/embed/UTygF2V0MtY" frameborder="0" allowfullscreen="">
</iframe>
------------------------------------------------------------------------
An amazing song, so many things stuffed into 4 minutes 10 seconds. Wow!
Tom Rush - Child's Song
<iframe width="420" height="315" src="//www.youtube.com/embed/8Dx6Zdx2AJ4" frameborder="0" allowfullscreen="">
</iframe>
------------------------------------------------------------------------
The Free Design - an amazing band from far far away and very long ago. Wikipedia says that his music is called 'Sunshine Pop' with elements of Jazz, part lullaby (for grown ups who still like Sesame Street); amazing;
<iframe width="420" height="315" src="//www.youtube.com/embed/vvLO1i0mTIo" frameborder="0" allowfullscreen="">
</iframe>
<iframe width="420" height="315" src="//www.youtube.com/embed/MUw8bxYgyTE" frameborder="0" allowfullscreen="">
</iframe>
------------------------------------------------------------------------
Best song ever. Renaissance - Carpet of the Sun.
<iframe width="420" height="315" src="//www.youtube.com/embed/XtSQFewjDD0" frameborder="0" allowfullscreen="">
</iframe>
------------------------------------------------------------------------
I think that the phrase "I'll keep you warm and you keep me sane" is very clever and deep one, it only appears in the lyrics of the "song Apeman:"http://www.azlyrics.com/lyrics/format/apeman.html
<iframe width="420" height="315" src="//www.youtube.com/embed/eEep67akIn4" frameborder="0" allowfullscreen="">
</iframe>
------------------------------------------------------------------------
A beautiful song [Who knows where the time goes - Fairport Convention](http://www.youtube.com/watch?v=n2xODjbfYw8)
<iframe width="420" height="315" src="http://www.youtube.com/embed/n2xODjbfYw8" frameborder="0" allowfullscreen="">
</iframe>
; Some see this band as part of the [Cantebury scene](http://en.wikipedia.org/wiki/Canterbury_scene) ([here](http://www.yourcanterbury.co.uk/leisure/fairport_convention_celebrate_45_years_in_the_business_1_1359660) it is mentioned as such) I find it all very amazing; they are not just banging on the instruments, I mean they are real musicians and poets, all in one ...
<br /><br />"Who knows where the time goes" by Sandy Denny<br /><br />Across the evening sky, all the birds are leaving<br />But how can they know it's time for them to go?<br />Before the winter fire, I will still be dreaming<br />I have no thought of time<br />For who knows where the time goes?<br />Who knows where the time goes?<br />Sad, deserted shore, your fickle friends are leaving<br />Ah, but then you know it's time for them to go<br />But I will still be here, I have no thought of leaving<br />I do not count the time<br />For who knows where the time goes?<br />Who knows where the time goes?<br />And I am not alone while my love is near me<br />I know it will be so until it's time to go<br />So come the storms of winter and then the birds in spring again<br />I have no fear of time<br />For who knows how my love grows?<br />And who knows where the time goes?
------------------------------------------------------------------------
<iframe width="420" height="315" src="//www.youtube.com/embed/EepdzqFZXNU" frameborder="0" allowfullscreen=""></iframe>
</p></blockquote>
the Bear watches the Dao
2013-05-04T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/05/04/post-57
["", "blogg"] | <notextile>the Bear watches the Dao</notextile>
<hr />
<p class="publish_date">
04 May 2013
</p>
<p><br /><br />Reading a very interesting book in Russian (Full text here: <a class="ot-anchor" href="http://www.mgimo.ru/files/7069/7069.pdf" rel="nofollow">http://www.mgimo.ru/files/7069/7069.pdf</a> )<br /><br />English translation: "The Bear Watches the Dragon: Russia's Perceptions of China and the Evolution of Russian-Chinese Relations Since the Eighteenth Century" by Alexander Lukin<br /><br /><a class="ot-anchor" href="http://www.amazon.com/Bear-Watches-Dragon-Perceptions-Russian-Chinese/dp/0765610256" rel="nofollow">http://www.amazon.com/Bear-Watches-Dragon-Perceptions-Russian-Chinese/dp/0765610256</a><br /><br />The way to understand your own culture is to look at foreign cultures: in this instance it is China, the civilization that stood apart for ages.<br /><br />What stunned me is that Tolstoy was very much influenced by Chinese though, especially by Laozi.<br />Here it is! The philosophical and political thought of Tolstoy had such a big influence on the Russian Silver age of literature; and this thought in turn was influenced by a source that was not cited at all in Tolstoy's main body of work.<br /><br />I think that Tolstoy probably knew that his Ideas would not have gained any attention if he were to advertise them as inspired by Chinese philosophy: in the 19th century the perception of China was as static and ossified; whereas only the West stood for progress.<br /><br />Page 100:<br /><br />"Для Л. Н. Толстого Китай был важным примером неинду-<br />стриального, ненасильственного, «непрогрессивного» и близкого <br />к природе образа жизни. Он был убежден, что «китайцы должны <br />оказать самое благотворное влияние на нас хотя бы уже своим <br />необыкновенным умением работать и на небольшом пространст-<br />ве земли добывать и растить больше и лучше, чем у нас на в де-<br />сять раз большем пространстве»215. Л. Н. Толстой всегда восхи-<br />щался китайскими мыслителями, участвовал в нескольких пере-<br />водах китайских классиков и даже хотел учить китайский язык216<br />. <br />Он особенно одобрял концепцию «неделания» («увэй»), провоз-<br />глашенную древнекитайским философом Лао Цзы, и использовал <br />ее в своей критике преклонения европейцев перед успехами нау-<br />ки, техники и интенсификации труда217. Русский писатель нахо-<br />дил и у Лао Цзы, и во многих других известных китайских изре-<br />чениях идеи, близкие к его собственным идеалам: жизни физиче-<br />ским трудом в единении с природой. 19 марта 1884 г. он <br />записывает в дневник: «Читал Конфуция — разумное объяснение <br />власти… было для меня откровением… Власть может быть нена-<br />силие, когда она признается как нравственно и разумно выс-<br />шее…»218 В книгу «Круг чтения», в которой Л. Н. Толстой собрал <br />высказывания различных философов и писателей всех времен и <br />народов, имевших, по его мнению, универсальное значение, <br />снабдив их своими комментариями, он включил много отрывков <br />из «Дао дэ цзин» и некоторых других китайских произведений219<br />. <br />В журнальных статьях он часто приводил Китай в качестве при-<br />мера аграрной цивилизации, представляющей альтернативу евро-<br />пейскому техническому прогрессу, по отношению к которому <br />он был настроен крайне критически"</p>
where are the programmers?
2013-05-04T00:00:00+00:00
http://mosermichael.github.com//all/blog/2013/05/04/post-48
["", "blog"] | <notextile>where are the programmers?</notextile>
<hr />
<p class="publish_date">
04 May 2013
</p>
<p><a class="ot-anchor" href="http://blogs.law.harvard.edu/philg/2013/05/01/why-isnt-there-a-glut-of-good-software-engineers/comment-page-2/#comment-197727" rel="nofollow">http://blogs.law.harvard.edu/philg/2013/05/01/why-isnt-there-a-glut-of-good-software-engineers/comment-page-2/#comment-197727</a><br /></p>
<p><a href="http://blogs.law.harvard.edu/philg/2013/05/01/why-isnt-there-a-glut-of-good-software-engineers/comment-page-2/#comment-197727">Interesting discussion</a> that asks the following question: Why aren’t there lots of people from third world countries who learned programming from the internet, and who could then flock to the first world as gastarbeiters?</p>
<p>Here are my comments:</p>
<blockquote>
<p>Phil, you site open source projects as reputation builders.
Do employers really value open source contributions? My experience is that management sees them as distractions; things that can steal valuable company time;
Other indications are that open source people are seen as not pliant enough; and people with too strong opinions of their own are not valued int the market.
In general I don’t know if I should hide my open source projects on the CV or not.</p>
</blockquote>
<blockquote>
<p>On a practical note; I think the open source movement comes with its culture of sharing and its own set of ideas; the whole thing came out of the Hippy movement (just look at the long beard of Mr. Stallman ;-)
Now with all things cultural there comes the same problem: it is hard to transplant a given culture into the third world; all third world countries come with their own culture, with their own system of values. a foreign culture can’t be simply transplanted into the third world, it can be adopted, but the process takes time.</p>
</blockquote>
<hr />
<p>Also: ‘long time programmer’ wrote, and I sort of agree with him:</p>
<blockquote>
<p>“I’ve been the software industry for 30 years, both technical and business roles.
There is a shortage of software engineers because the average career of a code writing engineer is short – less than 5 years.
Most hands on software work is crappy. Fix an obscure bug in legacy code. Write a data filter to bulk load reference data. Rewrite some commodity functions for the zillionth time because the managers don’t want code that wasn’t invented here.
Getting assignments that lead to career development is hard. In any large engineering team, the politics are rough. Getting the best work is like fighting a pack of wolves fighting over a deer. Only the strongest survive.
The way to survive is to have both top technical skills (especially the skill to learn new technology fast), and have top social skills. To use the wolf analogy, you must be able to bring down the biggest deer, and defend it from the others. Let the others have the bones (i.e., the crappy assignments). They’ll soon starve and leave. There is a shortage of software engineers because few people can mix the technical and social skills necessary to survive for long.”</p>
</blockquote>
<h3 id="politics-">Politics ?</h3>
<p>On a second thought: is politics/struggle for good assignment really that rough? Large chunk of it is being there early: for a Startup that means being among the first developers, for a larger project that means to be when it all starts. During the early cycle of a start up you usually have better choices and get the more interesting assignments; On the other hand surviving in a start up / choosing the right one to begin with are matters that require politics and analytical skills.</p>
<h3 id="attempt-to-see-the-larger-picture">Attempt to see the larger picture</h3>
<p>Here I am trying to understand the bigger picture, also trying to pick an optimal strategy.</p>
<p>Most organizations are structured as a hierarchy; you have divisions divided into departments divided into teams; each entity is dedicated to a specific and well defined aspect of the problem that the organization is supposed to solve. If a worker is embedded within this strict structure than this can lead to frequent down times; This centralized structure has its strength and weaknesses;</p>
<p>Strengths
* Its the only known way that can yield predictable and planned results
* This organizational division very much suits the <a href="http://en.wikipedia.org/wiki/Waterfall_model">waterfall design process</a> ; first requirement definition, then system design, then implementation, etc. Whatever they say, there are many competing methodologies, but that’s the way things are run if you work for BigCorp.
* Scalability: this system can grow into very large organizations; a flat model is harder to grow.
* Specialization: the system favors specialization; if one is assigned to a particular field one will have the time to become an expert in this field.
* Large shops have some degree of job security (as long as the business environment does not change and the shop is making a profit).</p>
<p>Weaknesses
* Problem with innovation: the system is not well suited for trying out new things with unknown chances of success; the system is averse to risks.
* Effectiveness: problems of integration between organizational units; the easiest problems are those that can be solved within a team; harder to solve problems that need cooperation of different teams; even harder to solve problems that involve different departments.
* uneven resources utilization: Some workers can face significant downtime, others may be over stressed.
* Much stress on lower level of management; those who still have to do with technical matters.</p>
<p>So, the strategy will probably depend upon your goals</p>
<ul>
<li>If one wants to have a large impact and then the preferred place will be a Start-up; Of course this assumes that a certain degree of instability is acceptable.</li>
<li>If one needs to feed a family and values stability then big corporation will be preferred;</li>
</ul>
<p>In any event one should try to increase his own value; both in terms of value for the current employee and the potential value in the job market;</p>
<h4 id="specialization-or-generalization-">Specialization or generalization ?</h4>
<p>As in most fields of endeavor: should a programmer try to have a general skill set, or should he be a narrow specialist?</p>
<p>I think that the hierarchy of big corporations encourages programmers to be narrow specialist; however people with wider horizons are very much in demand;</p>
<ul>
<li>Tasks that require integration of systems do value generalists</li>
<li>Some places have position of infrastructure developer; a good infrastructure specialist know a thing or two in a wide range of fields.</li>
</ul>
<p>I think that personal open source projects can be of great value here, these can keep me out of a narrow focus;</p>
<p>So again, three cheers for Open source!
(though better not mention that in public or put it on the CV)</p>
Biblical matters
2013-04-18T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/04/18/post-1
["", "blogg"] | <notextile>Biblical matters</notextile>
<hr />
<p class="publish_date">
18 April 2011
</p>
<p>Biblical matters</p>
<p>My comment to <a href="http://chronicle.com/article/Biblical-Blame-Shift/138457/">this discussion</a></p>
<blockquote>
<p>“the Old Testament, by discriminating between true and false religion, was responsible for ushering in unprecedented levels of historical violence … Assmann argues that biblical monotheism, as codified by the Pentateuch, disrupted the political and cultural stability of the ancient world by introducing the concept of “religious exclusivity”: that is, by claiming, as no belief system had previously, that its God was the one true God, and that, correspondingly, all other gods were false. “</p>
</blockquote>
<p>My response:</p>
<p>The historian/archaeologist Ian Morris argues that rulers prior to the first Millennium BC were claiming divine legitimacy [1]; If during the Bronze age a Pharaoh decides to kill all Jewish infants, then ordinary Egyptians would see this as an act of heaven. This means that religiously motivated totalitarianism as understood now was the norm, not the exception during this age and time.
It is strange that this notion would not be not known to a professional historian.</p>
<p>Also: there can’t be a distinction between political and religious conflicts if the rulers are gods or god-like; here any political conflict is a religious conflict as well. If ruler A pays tribute to ruler B then the gods of ruler A are subordinate to the gods of B; so much for “divine translatability” …</p>
<p>More so, Monoteism becomes a prerequisite for concepts such as morality and truth, Morality and individual responsibility can’t be combined with the notion that the ruler is your total owner; by rejecting the totalitarianism of the early civilizations (hochkulturen in german) Judaism indeed created the opportunity for a great Humanization of the Axial Age.</p>
<p>[1] Ian Morris ‘Why the west rules - for now’ page 278</p>
<hr />
<p>Also in a way it is not true that the kingdom of Israel stood apart from other nations; so it does not seem that ‘religious exclusivity’ was much of a problem, international relations is an area that inspires pragmatism.</p>
<p>the future king David was serving as a mercenary for the Philistines</p>
<p>The many wives of King Solomon had something to do with foreign policy.</p>
<p>King Ahab joined a wider coalition of kings in the <a href="http://en.wikipedia.org/wiki/Ahab#Battle_of_Qarqar">battle of Qarqar</a></p>
<p>King Jehu would <a href="http://en.wikipedia.org/wiki/File:ShalmaneserIII.jpg">bow before Shlamaneser III</a></p>
<p>King Pekah would ally with the Rezin, king of Aram against King Ahaz, who had Tiglat Pileser III on his side.</p>
<p>King Hoshea was installed as a client of Tiglat Pileser III</p>
<p>Also, I never understood why Jonah the prophet would have to prophesize to the inhabitants of Ninveh, of all places.</p>
<p>But what do I know …</p>
<p>Also of interest is this <a href="http://en.wikipedia.org/wiki/List_of_artifacts_significant_to_the_Bible">List of artifacts significant to the Bible</a></p>
Dissent lands you in trouble
2013-04-14T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/04/14/post-14
["", "blogg"] | <notextile>Dissent lands you in trouble</notextile>
<hr />
<p class="publish_date">
14 April 2013
</p>
<p>Is it only me who thinks that nowadays they could have run into problems with the lyrics?</p>
<p><a href="http://www.youtube.com/watch?v=OWyVPV2Q4RQ"">STEPPENWOLF - POWER PLAY</a></p>
<p>Some say that cold war period was sort of more liberal in terms of civil liberties (in the western hemisphere) than what we have now: <a href="http://www.tomdispatch.com/post/175638/tomgram%3A_jon_wiener%2C_ike%27s_dream%2C_obama%27s_reality/#more">Ike’s dream Obama’s reality</a> ; others regard the discrimination of women and minorities as more characteristic for this age and time; go figure.</p>
<p>One of the problems that they could have run into in our days is here <a href="http://www.madinamerica.com/2012/02/why-anti-authoritarians-are-diagnosed-as-mentally-ill/">Why Anti-Authoritarians are Diagnosed as Mentally Ill</a>
Of course there are children and adults with genuine problems that can be helped by medication; the problem that the article tries to point out is that the psychiatric definitions are covering a wide spectrum and are subject to interpretation; the result of this interpretation may over time result in repressive practices. (<a href="http://www.ft.com/intl/cms/s/2/25391482-c2f5-11e2-bbbd-00144feab7de.html#axzz2UFE11QWW">This essay</a> also argues that the DSM diagnostic bible is covering an growing range of disorders, and that things are more and more diffuse )</p>
<p>If society does not tolerate dissent, then dissent is approached/tackled by other means; these means may also involve psychology/psychiatry. ( On the other hand: fewer closed institutions, so no <a href="http://en.wikipedia.org/wiki/One_Flew_Over_the_Cuckoo's_Nest_(novel)">One Flew Over the Cuckoo’s Nest</a> )</p>
<p>I am speculating along these line: maybe back in the past there was a time when sort of every move was measured against the ideological enemy of the cold war; In these times there was a strong disincentive in the west against a wide definition of psychological disorders, namely the Soviet practice of supressing dissent by psychiatric means
<a href="http://en.wikipedia.org/wiki/Political_abuse_of_psychiatry_in_the_Soviet_Union"">Political abuse of psychiatry in the Soviet Union</a></p>
<p>repressive tool here too was a wide definition of a pathological condition
<a href="http://en.wikipedia.org/wiki/Sluggishly_progressing_schizophrenia">Sluggishly progressing schizophrenia</a></p>
<p>Now maybe this disincentive is gone, together with the Soviet Union;It might be that we West needs an Ideological enemy in order to uphold the torch of freedom, the case for moral high values. On an related/unrelated note: If one compares the practices of the former GDR with those of the former Soviet Union; in the GDR they had repressive practices down from elementary school; the express desire was to filter out undesirable personalities, so that those would not get into university and get an education; see the regime was very frightened of its intellectuals. Maybe with time such a approach/filter might be repeated in a less liberal age and time;</p>
<p>Disclaimer:
Not all of my screws are tight, but whose are ?
And yes, here are the lyrics of the song:</p>
<p>———————————————-<br /><br />"Power Play" by Steppenwolf<br /><br />What gives you the right? Hey you<br />To stand there and tell me what to do<br />Tell me who gave you the power?<br />To stop me from livin' like I do<br /><br />Remember if you plan to stay<br />Those who give can take away<br />Don't bite the hand that feeds you<br /><br />Just one time I'd like to be somewhere<br />Where none of your clever lies fill the air<br />I'm tired of your frozen smile and your voice of tin<br />Just might all gang up on you, turn the knob and do you in<br /><br />Remember if you plan to stay<br />Those who give can take away<br />Don't bite the hand that feeds you<br /><br />This never ending power play<br />'Tween jealous greed and vicious hate<br />Is grinding us like giant millstones<br />But it can't be our only fate<br />It's time we got our heads together<br />And let 'em know that we're awake<br /><br />Those in the dark, you know they're no longer blind<br />They're breakin' from your strangle hold on their minds<br />Those who can see don't need no one to cross the street<br />Be careful who you're pushin' round, they just might find you obsolete<br /><br />Remember if you plan to stay<br />Those who give can take away<br />Don't bite the hand that feeds you<br /><br />What gives you the right? Hey<br />To stand there and tell me what to do<br />Tell me who gave you the power?<br />To stop me from livin' like I do<br /><br />Remember if you plan to stay<br />Those who give can take away<br />Don't bite the hand that feeds you</p>
its all about the money
2013-04-13T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/04/13/post-34
["", "blogg"] | <notextile>its all about the money</notextile>
<hr />
<p class="publish_date">
13 April 2013
</p>
<p><br /><br />"There was no price ticker scrolling across our screens telling us what these bonds and derivatives we traded were worth. In fact, no one really knew what any of this stuff was worth. … What this meant, in its simplest form, is that these traders could buy bonds at the “market” price from intelligent hedge fund managers in NYC and sell this same crap at much higher levels to unsophisticated (but legally considered “sophisticated”) pension funds and insurance companies in middle America. What I discovered, quite starkly, is that the part of Wall Street that I worked in was simply transferring wealth from the less sophisticated investors, often teachers’ pension funds and factory workers’ retirement accounts, to the more sophisticated investors" <a class="ot-anchor" href="http://nickchirls.com/my-time-at-lehman" rel="nofollow">http://nickchirls.com/my-time-at-lehman</a><br /><br />My comments:<br /><br />>Why should retirement accounts get invested in anything but government bonds and index funds?<br /><br />Here (somewhere near the end) Mr. Blank says that this is what got the silicon valley rolling. When pension funds were allowed to invest, control of the valley switched from the military to the VC funds <a class="ot-anchor" href="http://www.youtube.com/watch?v=ZTC_RxWN_xo" rel="nofollow">Secret History of Silicon Valley</a><br />Probably another reason is that bonds and index funds do not yield enough to keep the pension funds going; people now live a longer life on average, so they need to pay more on each pension; so its all screwed up.<br /><br />Everything is screwed up; now that probably that has something to do with the fact that energy prices & commodities are high; there is less energy to go round, so other creative means are found to create 'wealth'; these tricks <b>increasingly</b> have something to do with extracting something from pocket A and transferring it to pocket B.<br /><br />(that's my comment here <a class="ot-anchor" href="https://news.ycombinator.com/item?id=5542231" rel="nofollow">https://news.ycombinator.com/item?id=5542231</a> )<br /><br />I also edited the wikipedia article on ZIRP (zero interest rate policy) lets see how long my change will stay :-) <a class="ot-anchor" href="http://en.wikipedia.org/wiki/Zero_interest-rate_policy" rel="nofollow">http://en.wikipedia.org/wiki/Zero_interest-rate_policy</a><br /><br />Chris Modica and Warren Sulmasy find that the ZIRP policy follows from the need to refinance a high US budget deficit and from the need to recapitalize the worlds banking system in the wake of the Financial crisis of 2007–2008 ( Chris Modica, Warren Sulmasy. "Why the Federal Reserve Bank Has a Near Zero Interest Rate Policy," YAHOO! Finance <a class="ot-anchor" href="http://finance.yahoo.com/news/why-federal-bank-near-zero-143000544.html" rel="nofollow">http://finance.yahoo.com/news/why-federal-bank-near-zero-143000544.html</a> )</p>
the computer that answered 42
2013-04-10T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/04/10/post-1
["", "blogg"] | <notextile>the computer that answered 42</notextile>
<hr />
<p class="publish_date">
04 April 2013
</p>
<p>SAGE - an air defense command and control system of the 50ies; in order to do it they had to invent the first computer network; first real time system, first high availability/redundant system. It was obsolete when it was completed, because ICBMs were so much more dangerous than bombers, but in the process they learned to engineer complex computer systems.</p>
<p><a href="http://scottlocklin.wordpress.com/2013/03/28/the-largest-computer-ever-built/">largest computer ever built</a></p>
<p>… and IBM got the know how for the S360 system by doing the SAGE contract; the whole system cost 10 billion $ ; Ha, now I know why Eisenhower was so upset about the <a href="http://www.youtube.com/watch?v=8y06NSBBRtY">military industrial complex</a></p>
<p>Another interesting talk on the subject: <a href="http://www.youtube.com/watch?v=ZTC_RxWN_xo">The Secret History of Silicon Valley</a></p>
<p>Another observation: IBM “Watson” AI system has sucessfully competed with humans at answering questions in a trivia show; congratulations: this is of a true advancement in the sorrow state of Artificial Intelligence; Now some observers see some tangible, though indirect profit for IBM here:
IBM makes a lot of money out of contracts for complex system that are paid for by tax payers money; if the firm is regarded as the body that can solve complex problems, then that alone puts IBM global solutions ahead when competing for government contracts.</p>
<p><a href="http://en.wikipedia.org/wiki/Watson_(computer)#Public_reaction">Watson - public reaction</a></p>
<p>It is also interesting that those very big companies are so fond of very big government contracts. The usual economic justification for big companies is <a href="http://en.wikipedia.org/wiki/Economies_of_scale">economies of scale</a>
I think that public contracts have a very high premimum, this premium is there to cover inefficiencies of big companies; Of course you need a big company to service a big contract, so there is a symbiotic relationship between big government and big corporations.</p>
Graph search
2013-03-25T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/03/25/post-25
["", "blogg"] | <notextile>Graph search</notextile>
<hr />
<p class="publish_date">
25 March 2013
</p>
<p><br /><br />How Graph Search is working (sort of) - this article is about the mechanism of executing the query<br /><a class="ot-anchor" href="http://www.facebook.com/note.php?note_id=10151361720763920" rel="nofollow">http://www.facebook.com/note.php?note_id=10151361720763920</a><br /><br />Even more interesting is this article on how natural language sentences are turned into queries<br /><a class="ot-anchor" href="https://www.facebook.com/note.php?note_id=10151432733048920" rel="nofollow">https://www.facebook.com/note.php?note_id=10151432733048920</a><br /><br />This system seems to be an example of a shallow parser <a class="ot-anchor" href="http://en.wikipedia.org/wiki/Shallow_parsing" rel="nofollow">http://en.wikipedia.org/wiki/Shallow_parsing</a><br /><br />All in all it seems to be an interesting development: search is now the frontier of classical Artificial Intelligence.</p>
readable Lisp
2013-03-21T00:00:00+00:00
http://mosermichael.github.com//all/blog/2013/03/21/post-55
["", "blog"] | <notextile>readable Lisp</notextile>
<hr />
<p class="publish_date">
21 March 2013
</p>
<p>Interesting project: they want to make Lisp usable by formatting it as if it where Python: the level of indentation would take the meaning of the level of bracket nesting. <a href="http://readable.sourceforge.net">The Readable S-Lisp expression project</a></p>
<p>Now some people like me can’t get used to pythonic indentation, but others might find that this is a great feature that could enable Lisp to jump into the 21st century.</p>
<h3 id="another-interesting-input-format">Another interesting input format</h3>
<p>The concepts of graphical programming languages so far has not been a great success, however here is an interesting combination of graphical elements and text based syntax ; the <a href="http://edutechwiki.unige.ch/en/Boxer">Boxer programming language</a> here the graphical element (the box) is used as a substitute for syntactic construct: drawn boxe are used instead of scope delimiters (begin … end , { .. } etc ).</p>
<p>This has the utility that there are fewer occasions for syntax error.<a href="http://www.soe.berkeley.edu/boxer/20reasons.pdf">A report on Boxer</a></p>
<p>I find that the subject of alternative input formats is a very interesting one. I did not play with the Boxer language, however there is the challenge of writing a good editor that combines graphic features and text; also one needs a good debugger; etc. etc. etc. It is hard to implement a system that has both a great language and a great graphic user interface.</p>
the price of freedom
2013-03-20T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2013/03/20/post-61
["", "ramblings"] | <notextile>the price of freedom</notextile>
<hr />
<p class="publish_date">
20 March 2013
</p>
<p>Interesting <a href="http://blogs.law.harvard.edu/philg/2013/03/07/how-long-would-slavery-have-lasted-in-the-south-if-not-for-lincoln">discussion</a> : PhilG asks the question of “How long would slavery have lasted in the South if the confederate states had been able to leave the union?” - he argues that the abolition of slavery would eventually have happened without the terrible cost of the civil war, just like it did happen all over the world.</p>
<p>I would like to quote the following opinions here:</p>
<p>Quoting JamesJ:</p>
<blockquote>
.... I don’t think the war was necessary, the industrial revolution was eliminating the need for slaves ,
and was steamroller that would have eventually changed the South as it did everything else. The epic battle that Lincoln fought to save the Union gave the Federal Government the
power over States’ rights we have never recovered from, and while he may have freed the slaves, he enslaved us to the Federal Government,
which now is into almost every aspect of our lives, and puts down any dissent with quick brutal efficiency.
</blockquote>
<p>Quoting Brian Gulino:</p>
<blockquote>
I am struck by how appealing it is to have an underclass. Men in societies where women are oppressed resist progressive movements involving women’s rights. It’s nice to have half the population serve you. We in California enjoy the services of diligent and hardworking Latin Americans, working at reduced wages due to being illegal. We are loathe to give this up. And our tech company executives flock to Washington to insure a supply of intelligent Asians who can only remain in the U.S. at their behest.
Then we come to slavery, the crack cocaine of human exploitation. How wonderful it must have been to own slaves. We never would have given it up.
</blockquote>
<h3 id="now-come-my-2-cents">Now come my 2 cents</h3>
<p>by the way a little known fact is that most slaves ever … live now! ( some 30 millions, mostly in places like India, China, Africa and the former Soviet Union; <a href="http://in.reuters.com/article/2013/10/16/slavery-index-idINDEE99F0DN20131016">here</a> and <a href="http://www.globalslaveryindex.org/report/">here</a> ;
People can just get kidnapped and then spend the rest of their live as slaves; just like that.</p>
<h3 id="now-back-to-history">Now back to history</h3>
<p>Was it really the steam engine that delivered us from slavery? Lets look at a book that studies this subject. Welcome to <a href="https://archive.org/details/capitalismandsla033027mbp">Capitalism and Slavery</a> by Eric Williams; the book studies the forces that brought about the abolition of slavery in Britain. As it turns out the really important factor here were shifting economic interests, not moral outrage of Quakers.</p>
<p>I guess a similar thing happened in the USA; demand of cotton soared during early 18th century due to the industrial revolution (see steam engine and Manchester) . so by the 1850ies the USA was at a crossroads: either continue to subsidize the cotton farms by maintaining slavery - all at a great cost to society.
The South could have remained a raw material producer like South America, entirely dependent on one product - raw cotton. Further down the road they would have suffered when the economic focus of the home country changed; or when faced with competing producers.(BTW that’s what happened earlier with the British West Indies);</p>
<p>I guess the debate was also about what would have been the character of new states gained by westward expansion.</p>
<p>The other option was to turnaround and do other things, here slavery would have had to be abolished in order to break the vested interest of plantation owners.
The choice for the USA was: join the industrial revolution or go bust eventually.</p>
<p>So the lesson here is that moral outrage is probably not enough to bring about change in vested interest, one has to search for the real causes. Also the steam engine did not change things directly, it brought about change by altering the environment.</p>
<p>The issue of slavery appears throughout human history, each time in a new disguise; In the 20th century there were the slaves of the Soviet Gulag and slaves of fascist concentration camps; never forget.</p>
<h4 id="synopsis-of-capitalism-and-slavery-by-eric-williams">Synopsis of ‘Capitalism and Slavery’ by Eric Williams:</h4>
<p>Now comes a synopsis of the book, as much as I (mis)understood it.
Read the book <a href="https://archive.org/details/capitalismandsla033027mbp">here</a></p>
<hr />
<p>Adam Smith: the prosperity of a new colony depends upon one simple economic factor “plenty of good land.” Poor immigrants tended to run off on their small subsistent farms (earth-scratching); big farms had nobody to till the land.
Here comes slavery.</p>
<p>“Adam Smith thereby treated as an abstract proposition what is a specific question of time, place, labor and soil. The economic superiority of free hired labor over slave is obvious even to the slave owner. Slave labor is given reluctantly, it is unskilful, it lacks versatility”</p>
<p>“The reasons for slavery, wrote Gibbon Wakefield, “are not moral, but economical circumstances; they relate not to vice and virtue, but to production.” 13 With the limited population of Europe in the sixteenth century, the free laborers necessary to cultivate the staple crops of sugar, tobacco and cotton in the New World could not have been supplied in quantities adequate to permit large-scale production”</p>
<p>New England had diversified agriculture; so no slaves as there were no big cotton fields.
Sugar, tobacco and cotton need large farms and an army of cheap labor
Also they argue once there are enough hands around there is no need for slaves (?)</p>
<p>Lack of skill leads to depletion of land, so it creates a need for territorial expansion.</p>
<p>First slaves were Indian; not good for labor intensive crops like sugar or cotton;</p>
<p>Then came the whites ‘slaves’ - as means to pay for the trip (indentured servants) or criminals (this kind of slavery was a temporary contract!). It has been estimated that more than a quarter of a million persons were of this class during the colonial period;
In GB kidnappings where common and the authorities did not prosecute that.
On the official side even minor crimes were punished with shipping off to colonies; so courts+authorities all worked togather here, all had a share of the spoils. Heat of the south? - no problem, whites can adjust as it turns out;</p>
<p>By start of 18th century economics theory shifted - previously mercantilism would aim to accumulate gold; later they would aim at promoting exports; and developing industry+employment. So they needed working hands in GB; previously the fear was of overpopulation, later that was passe. Previously England would try to get rid of excess population (Malthus) later economists regarded a numerous population as a source of wealth / the more the better.</p>
<p>So end of 17th century saw shift to black slaves. Blacks where slaves and worked the heavy stuff like sugar and cotton. Here slavery was property relation = not temporary contract. Children of white slaves where not slaves; unlike blacks; All of them perfected the machinery of slavery with whites as slave; with blacks they only reused the old mechanisms with more vigor. There were not enough white slaves available; and black slavers were cheaper for the job.</p>
<p>Could they have done without black slaves? Australia did! As a result labor costs are high, and they need protectionism and there is no industry.</p>
<h5 id="the-slave-trade">The Slave Trade</h5>
<ul>
<li>end of 1660 - end of English civil war; England expands into the slave business; (+ colonies in the Caribbean created demand)</li>
<li>1672 the Royal African company - a monopolist in the triangle trade: Britain<del><span style="text-align:right;">Africa, Britain</span></del>>America, West indies: manufactured good; Africa<del><span style="text-align:right;">America,West indies:slaves; America, West indies</span></del>>Britain: raw materials</li>
<li>The whole business was high risk: planting colonial crops can fail; slaves can die on the ship; politics between the colonial powers;</li>
</ul>
<!-- -->
<ul>
<li>Glorious revolution and the end of the Stuarts marks end of monopoly trading.</li>
<li>volume of slaves traded explodes with start of free trade
1680 - 1686: annually about 5000 slaves
1670 - 1679: annually about 18000 slaves</li>
<li>English traders supply Spanish and French colonies with slaves: this creates competition for ware</li>
<li>conflict of interest; slave trade has now its own interest.
Spain did not trade in slaves, they stuck to the old papal division: Portugal - Africa; Spain - America; also they did not have the capital for that !</li>
<li>the business of slavery accumulated much wealth into Britain;
Church was all for it; even opposed instruction of slaves to Christianity - they did not like a Sunday for slaves; also Quakers took part in the trade
Opposition to slavery came from American Quakers (who did not profit from the scheme)
All of Britain profited from the scheme: “The seventeenth and eighteenth centuries were the centuries of trade, as the nineteenth
century was the century of production. For Britain that trade was primarily the triangular trade.”; and its stimulus to industry
“The profits obtained provided one of the main streams of that accumulation of capital in England which financed the Industrial Revolution.”</li>
</ul>
<!-- -->
<ul>
<li>Ship building: Britain’s sea might came from a build up of the trading fleet for the triangular trade; from 1705 to 1795 tonnage quadrupled; number of ships trebled;</li>
<li>Britain’s port towns: Bristol; Liverpool; Glasgow all grew, very much due to the slave trade + trade with colonies; Manchester grew due to increased demand, increased production.
Navigation act of 1651 allowed only English ships to trade with England; colonies were allowed to ship to/import from England only.</li>
</ul>
<!-- -->
<ul>
<li>The colonies were a great export market for woolen clothes (and later cotton) made in England.</li>
<li>with cotton clothes India had better wares that fared well in Africa; but England restricted their sale;</li>
</ul>
<!-- -->
<ul>
<li>Sugar refining: in 1615 England prohibited sugar refining in colonies, sugar imports to Britain carried a heavy duty; (this gave rise to conflicts of interest between raw material producers and sugar refineries) it had to be done in England; Bristol and Glasgow were big into sugar refinery;</li>
</ul>
<!-- -->
<ul>
<li>Surprise: rum was produced in the colonies and exported into Britain ; *this was allowed as not to ruin the colonies;
Also was used to at recruiting slaves / deals with slave dealers (get them drunk)</li>
</ul>
<!-- -->
<ul>
<li>Metals: need iron chains for the slaves; guns were exchanged for slaves (one gun for a slave)
The expansion of heavy industry was directly financed by means of triangular trade;
James Watt was financed by West India trade !</li>
</ul>
<!-- -->
<ul>
<li>Plantation owners got rich and then moved to Britain; the plantation was run by caretakers.
In Britain the plantation owners; merchants; suppliers all formed a powerful interest group that was against abolition of slavery.
(weight of influence that this group yields becomes apparent during American Revolution)</li>
<li>Many banks were created out of the need to finance the triangular trade project; often there was a progression from trader to merchant, and then to banker; The triangular trade significantly stimulated overall development</li>
</ul>
<!-- -->
<ul>
<li>(Wikipedia) Mercantilism: system in place from 16th to 18th century; the aim of the state was to accumulate as much gold (bullion) as possible;
it had the following components:
<strong>** encourage exports (with subsidies); discourage imports by means of high tariffs; restrict domestic consumption
**</strong> encourage production & agriculture
<strong>** encourage large working population toiling at low wages; keep prices high as to maximize profit
**</strong> grab colonies and restrict them their trade and what they are allowed to produce;
Views economics as zero sum game: my gain is your loss.</li>
<li>at and earlier stages Mercantilism made sense! (Adam Smith would not have liked to acknowledge that) It did:
<strong>** mobilized resources needed in conflict of England’s against Spain
**</strong> protected inefficient/growing industry (all industrial nations had initial long periods of protectionism (me))
<em>**</em> money supply was limited (all gold); so more money enabled bigger operations; also trade with China and India meant that money was flowing out!</li>
</ul>
<!-- -->
<ul>
<li>During early period the system was monopolistic: a few benefited from monopolies and trading concessions; this had its inefficiencies: monopolies breed rent seeking</li>
<li>Regulation by means of monopolies created bleak markets and smuggling (colonies who defied trading restrictions); brutal suppression of “lower classes”</li>
</ul>
<!-- -->
<ul>
<li>Mercantilism ended with the industrial revolution: shift from trade to production happened;</li>
<li>trading is zero sum activity (limits how much one can take advantage of the other side); production: the more you can make and sell, the better (no ceiling to growth)
Also Britain could not argue for other markets to open up to its industry while keeping protectionist measures;</li>
<li>Britain’s power shift: after the ‘glorious revolution’ power shifted to parliament; Monarch had an easy job granting concessions; in parliament this is more complicated.</li>
<li>Adam Smith and the Wealth of Nations / the American Revolution : all factors that lead to scrapping of mercantilism</li>
<li>American revolution “it stimulated that growing feeling of disgust with the colonial system which Adam Smith was voicing”</li>
<li>Q: Which factor was more important? Hard to tell.</li>
</ul>
<h5 id="american-revolution">American Revolution:</h5>
<p>New England and continental colonies made a fortune out of exporting stuff to West Indies (agricultural output - wheat, dried fish; horses): also now they had regional specialization: Carribean - sugar,tobacco ; continental colonies - food for West Indies; this created conflict: Britain’s agriculture could not export to West indies; but so the colonies got the money to pay for British manufactured goods, this was more important.</p>
<ul>
<li>British plantation owners were against acquiring new Caribbean colonies: they were afraid that increased supply would drive down the prices; sugar prices were kept high - even though French colonies increased production. Also after seven year war, Britain returned Cuba and Guadeloupe to France,and got back Canada and Florida - all because of plantation owners interest !</li>
<li>Colonies demanded back payment in gold; now that the prices were no longer real, this was very bad for the plantation owners; Also the colonies would trade with the French realm; this put them into conflict with Britain (or rather the class of plantation owners that had the say)</li>
<li>1733: Molasses act; 1764 Sugar (duties) act ; both prohibited/restricted French colonies from selling molasses (raw material for Rum) to the continental colonies; Now that was important, as Rum was a major way to pay Britain for its wares; now this backlash lead to the American Revolution</li>
<li>Q: Was the American revolution caused by Tom Pain or was it due to trade conflicts? Hard to tell.</li>
</ul>
<p>The American Revolution lead to serious disruptions: West Indies went hungry; 15000 slaves died in Jamaica of hunger 1780-1787; USA was prohibited from trading with West Indies - because of navigation act; now they did anyway - this raised prices of US goods.</p>
<ul>
<li>USA became transport carrier for Spain and France - Britain sacked navies of those countries ;-)</li>
<li>The West Indies were further bleeding - further decline sets in; also had to compete with French colonies where Sugar was cheaper;</li>
<li>Now the Caribbean was lost to Britain - so Imperial focus switched to India (later China) (poor Indians).</li>
</ul>
<!-- -->
<ul>
<li>After 1787 Saint Dominique (French) started to produce lots of sugar; so Jamaica and Barbados (British) could not compete on price;</li>
<li>PM William Pitts plan: start sugar production in East India and abolish slavery, so as to ruin the French colonies - not feasible due to opposition of West Indies mafia.</li>
<li>After the French revolution Saint Dominique tried to join Britain (!) ; Britain agreed but the British army failed to take it. (actually this drove sugar prices up, at least temporarily)</li>
<li>If Saint Dominique would have become British, then Britain would have scrapped the plan to abolish slavery !</li>
</ul>
<!-- -->
<ul>
<li>Napoleon’s continental blockade started; British West indies could not sell sugar to Europe; so West India operated at a loss;</li>
<li>Britain decided to limit its production of sugar; so 1807 saw the abolition of the slave trade (less slave -> less sugar production)</li>
<li>Abolition of the slave trade as a means to ‘downsize’ the losses of sugar islands;</li>
<li>(the book does not mention it: due to the continental blockade Europe started to make sugar out of Sugar beets)</li>
<li>Also Mauritius, Cuba, Brazil started to plant sugar cane; so British colonies lost further weight)</li>
<li>So by 1833 the abolition of slavery was the means to get to ‘downsize’ loosing sugar islands; and to dismantle the sugar monopoly granted to West Indian colonies;</li>
<li>The decline and loss of direct interest in West Indian colonies was the source for British abolition of slavery - not moral outrage of the Quakers;</li>
<li>Independence demolished the concessions/monopolies - the basis of mercantilism; so actually it did Britain a favor ;</li>
<li>Also the industrial revolution #1 started to take off; so the colonial system / triangular trade was no longer relevant. (“In the age of mercantilism Manchester was Liverpool’s hinterland, in the age of laissez faire Liverpool was Manchester’s suburb”)</li>
<li>1807 - slave trade abolished; 1833 - slavery abolished; 1846 - preferential sugar duties abolished; The West Indies were nailed;</li>
<li>The corn laws (protecting British agriculture) were repealed because lower grain prices could help at keeping wages low; and that was now more important</li>
<li>by 182ies: centers of abolitionist movement: Birmingham (iron), Sheffield (steel) Yorkshire (wool); All of them out of a mix of mercantile and ethical opposition to slaver; Even Liverpool and Glasgow turned against West Indian monopolies - for free trade (apparently ignorant of who was gathering cotton and tobacco). Sugar refineries were opposed to West Indian monopolies - they were limiting production and driving up the prices; Shipping was for free trade (more to transport) but against repeal of Navigation laws.</li>
<li>“This was going too fast for a government still dominated, in 1832, by the landed aristocracy and therefore sympathetic to its colonial brethren. The government adopted a temporary compromise. In return for emancipation, the right of the West
Indians to the monopoly of the home market was confirmed, whilst the unrestricted importation of foreign sugar was permitted but only for refining and export to Europe.”</li>
<li>Brazil could not do without slaves; so “The British capitalists, therefore, began a vigorous campaign against their government’s policy of forcible suppression of the slave trade by stationing warships on the African coast.” (but for free trade they were)</li>
<li>Abolitionists initially did not advocate emancipation; they were against the slave trade; Only from 1823 did they adopt the aim of emancipation - but they demanded ‘gradual transformation’; only by 1830 did they demand immediate abolition - again they selectively opposed American slavery, and did not push the issue of East Indian slaves, and not too consistent with respect to Brazilian/US/;Cuban slavery; abolitionists were social conservatives allied with Capitalists / serving the new owner class interests</li>
</ul>
<h5 id="meanwhile-far-away-from-britain---in-the-colonies">Meanwhile: far away from Britain - in the colonies:</h5>
<p>In 1823 Britain began a limited reform program in Trinidad and British Guinea: abolition of the whip; nine hour working day (!); freedom for slaves born after 1823; appointment of ombudsman for slaves; savings bank for slaves; evidence of slaves heard in court : Plantation owners were very much opposed to these ‘concessions’ - talks of secession started , but ‘colored’ people where now loyal to Britain.</p>
<p>The slaves thought that after the abolition of the slave trade that Britain has declared emancipation, but that this has been withheld by their local governor. Each change of governor was understood as sign of coming emancipation ; growing slave revolts after 1800 ; by 1833 there was the clear choice between revolution or emancipation from above.</p>
<h5 id="conclusions">Conclusions</h5>
<p>“Men, pursuing their interests, are rarely aware of the ultimate results of their activity.
The commercial capitalism of the eighteenth century developed the wealth of Europe by means of slavery and monopoly. But in doing so it helped to create the industrial capitalism of the nineteenth century, which turned round and destroyed the power of commercial capitalism, slavery, and all its works”</p>
<p>Actors act on self interest, but do not see long term consequences; they did not foresee that free trade would be better than the colonial system !</p>
<p>Moral ideas of an age have to be understood in context with economic realities / interests. “…But historians, writing a hundred years after, have no excuse for continuing to wrap the real interests in confusion.”</p>
<p>The influence of the West Indian lobby in britain “can only be explained by the powerful services it had previously rendered and the entrenchment previously gained”</p>
<p>Prejudice usually serves the purpose of some interest; however prejudice remain - even when this interest is not longer relevant.</p>
a bold project
2013-03-14T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/03/14/post-1
["", "blogg"] | <notextile>a bold project</notextile>
<hr />
<p class="publish_date">
14 March 2013
</p>
<p>Oooo! Object oriented programming in Bash shell; what a bold project.</p>
<p><a href="http://sourceforge.net/p/oobash/code/937/tree/oobash/docs/">oobash project</a></p>
<p>every programming language can have its own object system (maybe with the exception for qbasic :-)</p>
<p>Lisp has the common lisp object system (<a href="http://en.wikipedia.org/wiki/Common_Lisp_Object_System">CLOS</a> ) of course these are implemented as Lisp macros;</p>
<hr />
<p>some people wonder if C** is worth the effort ( <a href="http://yosefk.com/c++fqa/">C** FQA</a> )</p>
<p>One can always do Objects in C with abstract data types and function pointers instead of virtual functions; i suspect that the real ‘killer feature’ of C** is that one can do objects with fewer keystrokes; the other ‘killer feature’ is that the syntax seems familiar to C programmers, however this is deceptive, learning C** well takes a lot of time…</p>
<p>C** templates are sort of awful; D seems to have a better template facility ( <a href="http://dlang.org/template.html">link</a> )</p>
<p>another big gripe about C** are header files - the feature inherited from C; every type or function defintion must be defined before use, so one has to include the text of the definition into the currently compiled source file; One of the nicer points of Java (and C#) as a language is that it gets rid of header files;</p>
<p>No that is because of the preprocessor; with C it gathers one source file with all included headers into one text, compiles it and type checks everything against the intermediate representation;</p>
<p>Java and friends compile one file as is and do type checking of what is known locally in that file, if there is a call to a method of an object that is not known from given information then that call is not type checked at this stage; Once all other files of a package have been compiled then it can perform type checking of calls from objects declared in the same package; the object signature lookup is conveniently done from class file - the same format that is used to run the stuff (but you can do without that like in D)</p>
<p>So that’s another major point where backward compatibility with C is not quite helpful.</p>
Prohibition might have a point ...
2013-03-12T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/03/12/post-1
["", "blogg"] | <h2 id="prohibition-might-have-a-point-">Prohibition might have a point …</h2>
<p class="publish_date">
11 March 2013
</p>
<p>My comment to this article was not published, so I post it here:
<a href="http://thehumanist.org/march-april-2013/prohibition-humanism/">article in question</a>
The article argues for the legalization of drugs, I don’t think thats a good idea.</p>
<hr />
<p>What about substance induced psychosis?
Why is this problem always ignored in this kind of discussions ?</p>
<p><a href="http://en.wikipedia.org/wiki/Substance-induced_psychosis">Substance induced psychosis</a></p>
<p>The war on drugs might be a failure; but the cost of having
to deal with psychotic / schizophrenic patients might well blow up the health system. Aren’t you willfully misguiding the public by ignoring this problem?”</p>
<hr />
<p>From: PSYCHOSIS WITH COEXISTING SUBSTANCE MISUSE ASSESSMENT AND MANAGEMENT IN ADULTS AND YOUNG PEOPLE / National Clinical Guideline Number 120” <a href="http://www.nice.org.uk/nicemedia/live/13414/53691/53691.pdf">Link</a></p>
<blockquote>
<p>To date, no single UK study has attempted to estimate the combined total healthcare and societal costs of treating people with psychosis and coexisting substance misuse. In 2007, the total health service costs of severe mental illness (schizophrenia, bipolar disorder and related conditions) were estimated at £3.8 billion while the total costs of lost employment were estimated at £5.4 billion (McCrone et al., 2008). Based on UK-based estimates of prevalence rates of between 36 and 44% for people with coexisting substance misuse (Menezes et al., 1996; Weaver et al., 2003), it is possible that the total annual health service and productivity costs of psychosis and substance misuse could be between £3.3 and £4 billion. However, further empirical research is required to assess the true economic burden of psychosis and substance misuse in the UK.</p>
</blockquote>
<hr />
<p>It turns out that British English is a bit confused about numbers; so I don’t know how much money they are talking about:</p>
<p>From <a href="http://oxforddictionaries.com/words/how-many-is-a-billion">Oxford Dictionaries</a></p>
<blockquote>
<p>In British English, a billion used to be equivalent to a million million (i.e. 1,000,000,000,000), while in American English it has always equated to a thousand million (i.e. 1,000,000,000). British English has now adopted the American figure, though, so that a billion equals a thousand million in both varieties of English.</p>
</blockquote>
<p>But it sounds like a lot of money, and legalization of pot would not help with that figure.</p>
<hr />
<p>The Chinese have fought two wars against the West over the <a href="http://en.wikipedia.org/wiki/Opium_Wars">prohibition of Opium</a> ; China lost resulting in what it regards as the <a href="http://en.wikipedia.org/wiki/Century_of_humiliation">Century of Humiliation</a> .
So the health issues involved probably have some wider implications, besides the cost of treatment.</p>
Can virtualization improve Security?
2013-03-11T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/03/11/post-10
["", "blogg"] | <notextile>Can virtualization improve Security?</notextile>
<hr />
<p class="publish_date">
11 March 2013
</p>
<p><br /><br /><a class="ot-anchor" href="http://www.schneier.com/blog/archives/2013/03/phishing_has_go.html" rel="nofollow">http://www.schneier.com/blog/archives/2013/03/phishing_has_go.html</a><br /><br />Now assuming you have a hardened OS (no listening services); and you run the web browser from a Virtual machine hosted by the OS; now given that they hack the WEB browser; how hard is it to hack out of the VM into the real OS ???????<br /><br />Can't we use hardware virtualization as a tool that divides the computer realm into compartments; much like a ship; with one compartment down the ship is still afloat (even spaceships do that now ;-)<br /><br />I guess if enough people will do this trick then they will find a way to break it …. But until that happens, it will pass as a very good security measure.<br /><br />A further addition: all outgoing packets from the VM will be marked by vlan tagging; you can't change that from within the VM; so you can't break out of your little sub network.</p>
modern times
2013-03-11T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/03/11/post-1
["", "blogg"] | <h2 id="modern-times">modern times</h2>
<p class="publish_date">
11 March 2013
</p>
<p>The UK has <a href="http://www.bbc.co.uk/news/uk-22007058">seven social classes</a> that’s what I call progress - the Indian caste system has only five castes.</p>
<p>Interesting facts about <a href="http://en.wikipedia.org/wiki/Social_mobility#Country_compariso">social mobility</a></p>
<p>In the US and UK there is a low level of social mobility - if you are rich then your kids will be rich;</p>
<p>Also: in countries with a high level of income inequality there is lower degree of social mobility.
<a href="http://en.wikipedia.org/wiki/File:The_Great_Gatsby_Curve.png">The graph that says this</a></p>
<p>Now social inequality tends to sharpen over time ; maybe the exception here are a few socially oriented countries in northern Europe, but the tendency exists. This development leads to a lesser degree of social mobility, and inevitably to ingrained inequalities of the education system.</p>
<p>After all its the education system that perpetuates inequalities; See university tuition fees: <a href="http://www.deseretnews.com/article/700213608/College-costs-skyrocketing-since-1970s.html?pg=all">From 1982 to 2007, the national average increase at America’s universities was 400 percent</a></p>
<p>All these tendencies togather can lead to a weakening of democracy and to curtailment of civil liberties.</p>
Open source ebook readers
2013-03-10T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/03/10/post-45
["", "blogg"] | <notextile>Open source ebook readers</notextile>
<hr />
<p class="publish_date">
10 March 2013
</p>
<p><br /><br /><a class="ot-anchor" href="http://fbreader.org/" rel="nofollow">http://fbreader.org/</a> - slower, best user interface (can click on items in table of contents)<br /><br /><a class="ot-anchor" href="http://blog.kowalczyk.info/software/sumatrapdf/free-pdf-reader.html" rel="nofollow">http://blog.kowalczyk.info/software/sumatrapdf/free-pdf-reader.html</a> - faster but less usable (can't click on items in table of contents)<br /><br />Also surprise, surprise, Adobe acrobat is not that bad as a book reader for PDF files ! In the View menu, select the 'Read Mode' option (or Ctrl+H), and voila !<br /><br />Now for years I was suffering from the default mode of viewing PDF files, which splits up the text into pages; Navigation between the pages is a bit edgy, so longer texts are hard to read that way. Interesting if other people have a similar experience - some useful feature hidden in a behemoth program, and this non obvious feature only makes the whole thing usable.</p>
The east/west business sort of matters
2013-03-07T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/03/07/post-58
["", "blogg"] | <notextile>The east/west business sort of matters</notextile>
<hr />
<p class="publish_date">
07 March 2013
</p>
<p><a href="http://www.amazon.com/Why-West-Rules-Now-Patterns/dp/0312611692">Why the west rules – for now</a></p>
<p>A very interesting book - it explains the big picture of history; how to interpret what is known about history in terms of the following metrics: Extraction of energy; Social complexity; ability to wage war; how to explain the ups and downs (i.e. periodic breakdown of society - like in the story Nightfall by I. Asimov ); <br /><br />Another Asimov analogy - this book reminds me of the kind of reasoning that was behind psychohistory - the fictional science in the Foundation series.</p>
<p>I would have thought from the title that the book is arrogant western crap; no it isn’t , just the opposite.</p>
<p>more on the fictional <a href="http://en.wikipedia.org/wiki/Psychohistory_(fictional)">Psychohistory</a></p>
<p>In another fascinating development, it turns out there is a new area of study that does mathematical modeling of historical dynamics <a href="http://en.wikipedia.org/wiki/Cliodynamics">Cliodynamics</a> . Science fiction is turning into science, fascinating.</p>
Nothing new under the sun
2013-03-07T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/03/07/post-43
["", "blogg"] | <notextile>Nothing new under the sun</notextile>
<hr />
<p class="publish_date">
07 March 2013
</p>
<p><br /><br />After a short break of 500 years; Spain invites back the Jews; <br /><br />"Anyone who could prove their Spanish Jewish origins, he (the justice minister) said, would be given Spanish nationality."<br /><br />Next: Will this step save the Spanish state from bankruptcy? Stay tuned!<br /><br /><a class="ot-anchor" href="http://www.bbc.co.uk/news/magazine-21631427" rel="nofollow">http://www.bbc.co.uk/news/magazine-21631427</a></p>
purpose of redesigns
2013-02-21T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/02/21/post-53
["", "blogg"] | <notextile>purpose of redesigns</notextile>
<hr />
<p class="publish_date">
21 February 2013
</p>
<p><br /><br />I suspect the purpose of the last years gmail redesigns was to make the product less usable on purpose; I guess they are paying a lot for storage of old data and abandoned accounts - and all that did not quite pay off with more targeted advertising; guess the click-through rate on gmail adds is low so they want to minimize costs. fewer users would mean less storage costs.<br /><br />And they have all sorts of transient user data from the chrome browser, the auto-completion thing phones back home; they are not obliged to keep this data, but it is good enough to create an online profile ….<br /><br />Bruce Schneier says that we are sort of vassals of the big internet companies; I guess that comes with an implied condition - if we got out stuff for free, then people are ready to trade in stuff like 'privacy' … things change when people feel that they don't get their share of the deal….<br /><br /><a class="ot-anchor" href="http://www.wired.com/opinion/2012/11/feudal-security/" rel="nofollow">http://www.wired.com/opinion/2012/11/feudal-security/</a><br /><br />Google doesn't want to do evil; what if a new Pharaoh will lead this great house and what if he decides differently? Suddenly one might see all sorts of companies exchanging user data, all in order to create a richer experience for the Customer and his friends …</p>
very old times
2013-02-20T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/02/20/post-66
["", "blogg"] | <notextile>very old times</notextile>
<hr />
<p class="publish_date">
20 February 2013
</p>
<p>They had <a href="http://www.bbc.co.uk/ahistoryoftheworld/objects/TnAQ0B8bQkSJzKZFWo6F-g">beer rations in Mesopotamia</a> - that raises a few questions: did they always have enough beer for the workers ? Could one trade his share for some other goods ? did they have to stand in line for their share ? What happens if they run out of beer ? What kind of society is that that has to mandate the consumption / distribution of stuff to its subjects ? If you run away from Mesopotamian rule and get into Egypt, how would things be different ?</p>
<p>Maybe <a href="http://en.wikipedia.org/wiki/Victor_Pelevin">Victor Pelevin</a> is not too far off when he likened Soviet / post Soviet society to Mesopotamia ? … and more intriguing - are these all questions unique to persons who grew up behind the Iron curtain ?</p>
<hr />
<p>Is rationing of food a feature of a high-end state? <a href="http://www.amazon.com/Why-West-Rules-Now-Patterns/dp/0312611692">Ian Morris says</a> that in high-end states there are no vassals and that everyone is subordinate to the state (no fiefdoms allowed). A high end state needs a standing army and needs a huge number bureaucrats ; Of course the financing of these projects requires the state to raise taxes. (see <a href="http://www.upenn.edu/almanac/v48/n28/AncientTaxes.html">Taxes in the ancient world</a> )</p>
<p>However this development is associate with King Tiglat Pileser the III of Assyria, who lived in the eight century BC;</p>
<p>Can it be that the Mesopotamia of 3000BC was not so much different from the Neo Assyrian Empire of 800BC ?</p>
<p>Ian Morris says that the change has been brought about by a worsening of the climate, suddenly it was all cold and dry. (<a href="http://www.amazon.com/Why-West-Rules-Now-Patterns/dp/0312611692">Why the west rules</a> on page 86</p>
<blockquote>
Climate change forced tough choices on Mesopotamians. They could bury their heads in the sand as it encroached on their fields and carry on as usual, but the price of doing nothing would be hunger, poverty, and perhaps starvation. Or they could migrate to regions less dependent on the monsoon; but it is no small thing for farmers to abandon their well-tended fields. In any case, the Hilly Flanks—the obvious place to go—was already packed with villages. In 2006 archaeologists at Tell Brak in northeast Syria uncovered two mass graves of young men dating to around 3800 BCE, apparently the victims of massacres. Moving back to the crowded, violent Hilly Flanks might not have been a very attractive option. If enough Mesopotamians had done nothing or run away, this new core would have collapsed. However, a third possibility presented itself. People could abandon their villages but stay in Mesopotamia, congregating in a few big sites. That seems counterintuitive: if crop yields are falling, cramming more people into smaller spaces should make things worse. But some Mesopotamians seem to have figured out that if more of them worked together they could run larger irrigation systems and store floodwaters until the crops were ready. They could feed more miners to dig copper from the ground; more smiths to make ornaments, weapons, and tools; and more traders to carry these goods around. So successful were they that by 3000 BCE bronze (an alloy of copper and a little tin) had largely replaced stone for weapons and most tools, sharply increasing fighters’ and workers’ effectiveness.
Getting to that point, though, required organization. Centralized administration was the answer. By 3300 BCE people were scratching onto little clay tablets such sophisticated records of their activities that most archaeologists call the symbols writing (even if as yet only a tiny scribal elite could read them). Little villages that could not support such sophisticated activities went to the wall while one site, Uruk, turned into a true city with maybe twenty thousand residents.
Mesopotamians were inventing management, meetings, and memoranda—the curses of life for so many of us today, and hardly the stuff of soaring narratives of human achievement. Yet as will become clear in the next few chapters, these were often the most important motors of social development. Organization turned villages in the Hilly Flanks and along the banks of the Yellow River into cities, states, and empires; failures of organization caused their fall. Managers are simultaneously the heroes and the villains of our story. The birth of management as the monsoons dried up must have been traumatic. We should probably picture bedraggled, defeated columns of the hungry slouching toward Uruk under a dusty sky, like Okies but without the jalopies, let alone the New Deal. We should probably also imagine angry villagers refusing to cede power to self-important bureaucrats who tried to requisition their fields or crops. Violence must often have been the outcome. Uruk could easily have broken apart; perhaps plenty of rival towns did.
We will never know the stories of the ancient managers who pulled Uruk through, but archaeologists suspect that they were tied to temples. Many pieces of evidence point this way, propping one another up like the poles in a tepee. For instance, excavations at temples have uncovered stacks of uniform-sized dishes known as “bevel-rimmed bowls,” probably for distributing food. The earliest clay tablets scratched with crude symbols come mostly from temples, and the symbol for “rations” on them is a sketch of a bevel-rimmed bowl. And when writing systems developed to the point they could record such information, they tell us that temples controlled broad acres of irrigated land and the labor to work them.
</blockquote>
<p>The pattern is that there is a certain pendulum that swings between high end states and low end ones; I don’t know what option would be preferable for an individual: what is preferable the violence of a centralized state or the whims of a vassal; the order and periodic terror of the state or the general chaos of medieval times?</p>
<p>The high end state seems to have had been a better option for the advancement of arts the sciences and technology: with a high production there is a greater chance that something trickles down to the thinkers; also a higher level of stability is good for the accretion of culture.</p>
<p>However when the state becomes repressive then most of these advantages are undone; so go figure
The thinkers of the <a href="http://en.wikipedia.org/wiki/Axial_Age">Axial age</a> where placed on the periphery, not in the big centers of civilization.</p>
<hr />
<p>Another idea: The Pharaohs claimed to be Gods but the rulers or Mesopotamia claimed to be God like; Some people like <a href="http://en.wikipedia.org/wiki/Erich_von_D%C3%A4niken">Daeniken</a> make a lot out of this distinction.</p>
<p>Can it be that this difference is due to the fact that in Mesopotamia power was usually local to the city state, in any event a subject could possible run away to a competing authority. So the power of the city state could not have been absolute; so there was no use to claim the impossible - it would be strange for a local leader to claim to be God when there is a competing leader next door to you.</p>
<p>In ancient Egypt was much more limited in this regard; there were fewer places to run to, for an individual the choices were severely limited, no use to question established believes.</p>
<hr />
<p>also interesting thing to note: Agriculture and irrigation must have been easier in Egypt – Mesopotamia had lots of problems with the irrigation project, the problem of soil becoming salin is much more severe than in Egypt, I guess that the irrigation system in Mesopotamia required more work. See <a href="http://mygeologypage.ucdavis.edu/cowen/~GEL115/115CH17oldirrigation.html">ancient irrigation</a>
So Mesopotamia was probably more technologically challenged than Egypt.</p>
<p>On the other hand Egypt had more political stability/continuity - which must have been good for the development / accretion of arts, science and technology.</p>
<p>Hard to tell which civilization was better off in the end.</p>
<hr />
<p>In any events there is a lot of room for speculation in these subjects. It is a very rewarding subject - it is also hard to say if one says something significant or if it is just rubbish.
I am not a historian, and the more I think on these subjects, the more confused I get …</p>
war games
2013-02-19T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/02/19/post-67
["", "blogg"] | <notextile>war games</notextile>
<hr />
<p class="publish_date">
19 February 2013
</p>
<p><br /><br />If a large meteorite were to have crashed into US/USSR during the early 80ies, it would have been interpreted as a nuclear attack; So thanks it did not happen. <br /><br /><a class="ot-anchor" href="http://en.wikipedia.org/wiki/Able_Archer_83" rel="nofollow">http://en.wikipedia.org/wiki/Able_Archer_83</a><br /><br />The meteor that crashed into the Urals made a bang worth 300 Kilotones of TNT; all that in a region that is stacked with strategic/nuclear shit; I shit my pants;<br /><a class="ot-anchor" href="http://english.ruvr.ru/2013_02_16/Meteor-caused-300-kiloton-blast-over-Urals/" rel="nofollow">http://english.ruvr.ru/2013_02_16/Meteor-caused-300-kiloton-blast-over-Urals/</a></p>
civic engagement
2013-02-16T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/02/16/post-11
["", "blogg"] | <notextile>civic engagement</notextile>
<hr />
<p class="publish_date">
16 February 2013
</p>
<p><br /><br />Interesting page where one can compare OECD countries by such indicators as health, education, salary, housing etc. etc. Did you know that Israel and Russia have the same level of civic engagement ? Both are close to zero ;-)<br /><br /><br /><a class="ot-anchor" href="http://www.oecdbetterlifeindex.org/countries/israel/" rel="nofollow">http://www.oecdbetterlifeindex.org/countries/israel/</a>
<br />
<br /></p>
<p>Why are other countries in this respect better off ? In the US people think that they can talk to a local representative, and that the representative is supposed to actually listen to what is being said.</p>
<p>Probably the system of local representatives suggests the notion that there is a well defined contact point to what is going on in higher places.</p>
<p>In Israel the member of Knesset / Parliament come from their respective party lists; so one needs to be part of a major political party in order to have a voice in serious matters, major parties tend to hold primary elections or something of that sort, minor parties don’t do that.</p>
<p>As for the listening part, Israelis have a problem with that. So there is no use in trying.</p>
<p>Or maybe it is the age of the country; democracy in other countries evolved over a very long time frame; maybe feedback mechanisms that allow for input from below are things that need time to grow. Even more so; these mechanisms are probably governed by custom and not by binding laws (Pooh Bear says Crustimoney Proseedcake); So lets examine the issue in another two or three hundred year …</p>
<p>Also: maybe some day a mixed system of local representatives vs party affiliated representatives might evolve here. Why did the founding fathers decide in favour of party based affiliation? Probably because of the maps: the question of boarders was not settled (as it is now), so there was no point in framing constituencies; but probably more important was that the voters were seriously divided along party lines of the various political movements, so an additional introduction of regional divisions would have been too much. The Nation is famous for its sectarianism: Must insert link to Monty Pythons <a href="http://www.youtube.com/watch?v=gb_qHP7VaZE">Peoples Front of Judea sketch</a></p>
<p>If in some distant future (rather some parallel universe, UNLESS <a href="http://www.jewishencyclopedia.com/articles/5849-eschatology">Final Redemption</a> comes sooner) the politics here might actually calm down and this issue could then be reconsidered.</p>
lots of fun
2013-02-15T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/02/15/post-38
["", "blogg"] | <notextile>lots of fun</notextile>
<hr />
<p class="publish_date">
15 February 2013
</p>
<p><br /><br />Oh, oh, A novel trick to achieve a comic effect; he is translating a text into basic English. This is Pure Genius. <br /><a class="ot-anchor" href="http://xkcd.com/1173/" rel="nofollow">http://xkcd.com/1173/</a><br /><br />Here is another one 'Up goer five'<br /><a class="ot-anchor" href="http://xkcd.com/1133/" rel="nofollow">http://xkcd.com/1133/</a><br /><br />Here is the 'Up goer five' editor - it tells you if a text uses more then the most common 1000 words.<br /><a class="ot-anchor" href="http://splasho.com/upgoer5/" rel="nofollow">http://splasho.com/upgoer5/</a><br /><br />More on Basic English<br /><a class="ot-anchor" href="http://en.wikipedia.org/wiki/Basic_English" rel="nofollow">http://en.wikipedia.org/wiki/Basic_English</a></p>
Кайф
2013-02-15T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/02/15/post-28
["", "blogg"] | <notextile>Кайф</notextile>
<hr />
<p class="publish_date">
15 February 2013
</p>
<p><br /><br />Кайф - После Октябрьской революции слово считалось относящимся к проявлению буржуазной идеологии и практически вышло из употребления.[2] Снова оно прошло в язык после VI Всемирного фестиваля молодёжи и студентов в Москве в 1957 году, когда стало модным «вворачивать» в разговоре слова иностранного происхождения.<br /><br /><a class="ot-anchor" href="http://ru.wikipedia.org/wiki/%D0%9A%D0%B0%D0%B9%D1%84" rel="nofollow">http://ru.wikipedia.org/wiki/%D0%9A%D0%B0%D0%B9%D1%84</a><br /><br />Гугл говорит что словом кайф стали пользоваться в 70ые годы<br /><br /><a class="ot-anchor" href="http://books.google.com/ngrams/graph?content=%D0%9A%D0%B0%D0%B9%D1%84&year_start=1800&year_end=2000&corpus=25&smoothing=3&share=" rel="nofollow">http://books.google.com/ngrams/graph?content=%D0%9A%D0%B0%D0%B9%D1%84&year_start=1800&year_end=2000&corpus=25&smoothing=3&share=</a><br /><br />Нет, слово Эйфория пишут чаще чем Кайф ! тоже иностранное однако. Трудно описать Кайф который испытывает общество начиная с 90ых годов ;-)<br /><br /><a class="ot-anchor" href="http://books.google.com/ngrams/graph?content=%D0%9A%D0%B0%D0%B9%D1%84%2C%D0%AD%D0%B9%D1%84%D0%BE%D1%80%D0%B8%D1%8F&year_start=1800&year_end=2000&corpus=25&smoothing=3&share=" rel="nofollow">http://books.google.com/ngrams/graph?content=%D0%9A%D0%B0%D0%B9%D1%84%2C%D0%AD%D0%B9%D1%84%D0%BE%D1%80%D0%B8%D1%8F&year_start=1800&year_end=2000&corpus=25&smoothing=3&share=</a><br /><br />Из "Доктора Живаго" мне запомнился следующий пассаж:<br /><br />"Он снова думал, что историю, то, что называется ходом истории, он представляет себе совсем не так, как принято, и ему она рисуется наподобие жизни растительного царства. Зимою под снегом оголенные прутья лиственного леса тощи и жалки, как волоски на старческой бородавке. Весной в несколько дней лес преображается, подымается до облаков, в его покрытых листьями дебрях можно затеряться, спрятаться. Это превращение достигается движением, по стремительности превосходящим движения животных, потому что животное не растет так быстро, как растение, и которого никогда нельзя подсмотреть. Лес не передвигается, мы не можем его накрыть, подстеречь за переменою места. Мы всегда застаем его в неподвижности. И в такой же неподвижности застигаем мы вечно растущую, вечно меняющуюся, неуследимую в своих превращениях жизнь общества, историю.<br /> Толстой не довел своей мысли до конца, когда отрицал роль зачинателей за Наполеоном, правителями, полководцами. Он думал именно то же самое, но не договорил этого со всею ясностью.<br /> Истории никто не делает, её не видно, как нельзя увидать, как трава растет. Войны, революции, цари, Робеспьеры это её органические возбудители, её бродильные дрожжи. Революции производят люди действенные, односторонние фанатики, гении самоограничения. Они в несколько часов или дней опрокидывают старый порядок. Перевороты длятся недели, много годы, а потом десятилетиями, веками поклоняются духу ограниченности, приведшей к перевороту, как святыне."</p>
спокойные времена
2013-02-15T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/02/15/post-27
["", "blogg"] | <notextile>спокойные времена</notextile>
<hr />
<p class="publish_date">
15 February 2013
</p>
<p><br /><br />Самые спокойные времена - это когда нет вопросов :-) когда возникают вопосы, времена становятся буйными. Еще наблюдение: все фигня по сравнению с Октябрьской революцией ;-)<br /><br />В "Докторе Живаго" слово "вопрос" встречается очень часто - и на самом деле начиная с четырнадцатого года появилось очень и очень много вопросов… потом в перестройку опять появились вопросы…<br /><br /><a class="ot-anchor" href="http://books.google.com/ngrams/graph?content=%D0%B2%D0%BE%D0%BF%D1%80%D0%BE%D1%81&year_start=1800&year_end=2000&corpus=25&smoothing=3&share=" rel="nofollow">http://books.google.com/ngrams/graph?content=%D0%B2%D0%BE%D0%BF%D1%80%D0%BE%D1%81&year_start=1800&year_end=2000&corpus=25&smoothing=3&share=</a></p>
detached observations
2013-02-11T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/02/11/post-16
["", "blogg"] | <notextile>detached observations</notextile>
<hr />
<p class="publish_date">
11 February 2013
</p>
<p><br /><br />your mind fails to see anything of importance, when you are standing on the top of the world, <br /><br />- Seems to be an original observation; at least google did not find it.</p>
a song of great political importance
2013-01-19T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/01/19/post-5
["", "blogg"] | <notextile>a song of great political importance</notextile>
<hr />
<p class="publish_date">
19 January 2013
</p>
<p><br /><br />This song reminds me of the upcoming election<br /><br />"So!<br />Come on ye childhood heroes!<br />Won't you rise up from the pages of your comic-books<br />your super crooks<br />and show us all the way.<br />Well! Make your will and testament. Won't you?<br />Join your local government.<br />We'll have Superman for president<br />let Robin save the day."<br />Jethro Tull<br /><br /><a class="ot-anchor" href="http://www.youtube.com/watch?v=M9JEPeeohYs&t=18m09s" rel="nofollow">Jethro Tull - Thick as a Brick full</a><br /><br />In the word of Janis Joplin: "This is a song of great political importance" ;-)</p>
Bailouts are not Kosher
2013-01-17T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2013/01/17/post-1
["", "blogg"] | <notextile>Bailouts are not Kosher</notextile>
<hr />
<p class="publish_date">
17 January 2013
</p>
<p>The Chief Rabbi of Israel says that bailouts are not Kosher and against Jewish Religious law. Business owners must be held responsible for misconduct.</p>
<p><a href="http://www.haaretz.com/jewish-world/jewish-world-news/israel-s-chief-rabbi-issues-unprecedented-halakhic-ruling-against-financial-misconduct.premium-1.494490">the story</a></p>
<p>This step does not change how the court views the issue, but it might make a moral difference for some businesspersons of Jewish descent. I think this statement required some courage on the part of the Chief Rabbi.</p>
<hr />
<p>I also added a comment:</p>
<p>I wish him well …
I hope that the powers that be will not try harm our wise Chief Rabbi, as our establishment tends to be a very spiteful one</p>
The Pooh programming language for kids and grown ups.
2012-12-11T00:00:00+00:00
http://mosermichael.github.com//all/pooh-lan/2012/12/11/pooh-intro
["", "pooh-lan"] | <h2 id="the-pooh-programming-language-for-kids-and-grown-ups">The Pooh programming language for kids and grown ups.</h2>
<p class="publish_date">
11 December 2012
</p>
<p>The pooh programming language is an educational programming language. I think this means that the language should help to overcome difficulties when one tries to learn it. Many other programming languages are full with tricky tricks that are there to save a few keystrokes while typing long programs, these aids are very powerful, but people who are not already used to thinking in complicated patterns are put off by them. Learning to program with a complicated language becomes some sort of rite for geeks, who already like complicated things; Pooh thinks that the language that a person uses is important ; it makes a person accustomed to think the way he speaks, this goes on until many people are no longer used to think in any other ways. The computer language one uses as a programmer therefore sets ones mind to thinking in geeky ways (probably only when programming?). This thought is also known as the <a href="http://en.wikipedia.org/wiki/Linguistic_relativity#Programming_languages">Sapir-Whorf hypothesis</a> ; maybe there is some truth in it with regards to computer languages. I think that a simple, yet powerful language would help here.</p>
<h3 id="availability">Availability</h3>
<p>Currently the project is available in source form only, It is released under the BSD Three clause license;
The copyright is owned by Michael Moser ( c ) 2012
Sources can be downloaded as <a href="https://github.com/MoserMichael/cstuff/zipball/master">zip archive</a>
or downloaded by the git program.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> git clone https://github.com/MoserMichael/cstuff.git
</code></pre></div></div>
<p>The project can be build on Linux or on Windows with Cygwin. To build the program,we need the presence of bison, flex, gcc and a recent version of make.</p>
<p>Build and install into /usr/local/ directory</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> cd build
./build-pooh.sh install
</code></pre></div></div>
<p>Build and install into /usr/alt directory</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> cd build
./build-pooh.sh install INSTALL_PREFIX=/usr/alt
</code></pre></div></div>
<h3 id="documentation">Documentation</h3>
<p>The <a href="/cstuff/docs/html/pooh/pooh-lan.pdf">Introduction to Pooh</a> which is not complicated. (currently still working on it :-(
Here <a href="/cstuff/docs/html/pooh/pooh-ref.pdf">Pooh library reference</a> which is at times short an more complicated
Here are <a href="/cstuff/docs/html/pooh/designnotes.htm">Some explanations of the design decisions</a> should be interesting for people who like programming languages.</p>
<h3 id="features">Features</h3>
<p>This section is interesting for people who already know a few programming languages.</p>
<ul>
<li>The interpreter can produce a detailed execution trace with with the annotated values for all variables. Here is an example to run a program that computes the factorial of the number 4.
!include tomd-include-0.txt</li>
<li>functions: all parameters are passed as <a href="http://en.wikipedia.org/wiki/Named_parameter">named parameters</a> , order of parameters does not matter. Named parameters make the program much more readable.</li>
<li>All values defined in a function are of local scope; if you want to access a global from a function, then you have to add the <b> global . </b> prefix to the variable name.</li>
<li>strings, numbers, arrays and hashes are not objects, one does not have to introduce object oriented programming from the start on;</li>
<li>A variable is defined when a value is assigned to a variable name;.</li>
<li>The compiler forbids to reference uninitialized values (like ‘use strict’ in Perl and recently optional in Javascript); also forbidden to call of undefined functions ; forbidden to use of array or hash as scalar. I really like Perl strict mode; too much dynamic typing gets me confused.</li>
<li>In arithmetic operators one can use both number and string variables (strings are converted automatically); Comparison operators - there is a set of numeric comparison operators, where strings are interpreted as numbers, and string comparison operators, where both numbers and strings are allowed and interpreted as string.</li>
<li>data structures: has vectors and hash values for collections.</li>
<li>A simplified model of references to values is supported.</li>
<li>Does object oriented programming by <a href="http://en.wikipedia.org/wiki/Prototype-based_programming">prototyping (prototype based programming)</a> ) ; access to data members via <b> this . </b> prefix to variable names</li>
<li>the concept of <a href="http://en.wikipedia.org/wiki/Closure_(computer_science)">closures</a> is supported, one can have objects via closures. Non local variables references have the <b> outer . </b> prefix.</li>
<li>the concept of <a href="http://en.wikipedia.org/wiki/Coroutine">co-routines / generators</a> is supported; for loops invoke a generator thread (similar to python)</li>
<li>has <a href="http://en.wikipedia.org/wiki/Higher-order_function">higher order functions</a> like map/foldl/filter/sort
<a href="http://rosettacode.org/wiki/Return_multiple_values">Multiple return values</a> / <a href="http://en.wikipedia.org/wiki/Assignment_(computer_science)#Parallel_assignment">Parallel assignment</a> is supported.</li>
<li>Strings are can span multiple lines and can embed expressions; very similar to like HERE documents; the language has one string constant format, not twenty.</li>
<li>It is possible to extend the language with shared libraries written in C.</li>
<li>It is possible to include other text, so it is compiled together with the program.</li>
<li>Compiler has detailed error messages. Invested quite some effort to make them clear.</li>
</ul>
<p>Things left out:</p>
<ul>
<li>No exceptions. Pooh programs either succeed or fail with a very detailed stack trace, its that simple.</li>
<li>No namespaces; another complication.</li>
<li>No meta programming; no pre-processor madness</li>
<li>No eval statement;</li>
<li>Currently no regular expression parsing; I would like something with fewer arcane rules, maybe BNF grammars like REBOL; Any ideas are and suggestions are really welcomed.</li>
<li>Currently no way to do unicode strings; Really Have To Fix That.</li>
<li>No auto-increment operators. One feature less to explain.</li>
</ul>
<p>All this means pooh is not strongly typed, and also not a dynamic language. Hope you are still interested ;-) Because of the simplified reference model, the language probably does not translate into efficient machine code. Don’t expect the current implementation to be fast, it is an AST walker; that’s because it is easier to do the trace output with an AST walker; also I want to add a <a href="http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">Read-eval-print loop</a> ( in a coming release.</p>
<h3 id="tests--examples">Tests / examples</h3>
<p>This table lists some example programs, they make up the test suite of the project and are run on each build.</p>
<table>
<thead>
<tr>
<th> </th>
<th> </th>
</tr>
</thead>
<tbody>
<tr>
<td><em>Test name</em></td>
<td><em>Description</em></td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/01-add.p.html">“01-add.p”</a></td>
<td>mixing strings with numers is allowed in arithmetic operators</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/01-copy-ref.p.html">“01-copy-ref.p”</a></td>
<td>with references, the := operator creates alias, left hand side refers to value returned by right hand side expression.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/01-function-call-by-ref.p.html">01-function-call-by-ref.p</a></td>
<td>function call by reference , call by value</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/01-hello-world.p.html">01-hello-world.p</a></td>
<td>hello world</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/02-fact.p.html">02-fact.p</a></td>
<td>function: get factorial of a number (recursive)</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/02-loop.p.html">02-loop.p</a></td>
<td>loops: function computes the sum of squares over integer range</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-int-range.p.html">03-int-range.p</a></td>
<td>function fills array with number range, array is the return value.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-matrix-mult.p.html">03-matrix-mult.p</a></td>
<td>test work with two dimensional matrixes.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-prime-while-loop.p.html">03-prime-while-loop.p</a></td>
<td>array test: sieve of Eratosthenes gets all prime numbers</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-print-array.p.html">03-print-array.p</a></td>
<td>array test: print all elements of array, iterate with while statement.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-print-table.p.html">03-print-table.p</a></td>
<td>iterate over table and print out values.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-print-table2.p.html">03-print-table2.p</a></td>
<td>iterate over table and print out values.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-qsort.p.html">03-qsort.p</a></td>
<td>a quick sort written in pooh</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-string-split.p.html">03-string-split.p</a></td>
<td>test split library function - string split into array.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-stringlib.p.html">03-stringlib.p</a></td>
<td>tests string library functions</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-table-any-kind-of-key.p.html">03-table-any-kind-of-key.p</a></td>
<td>table with any kind of keys</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/04-file-list.p.html">04-file-list.p</a></td>
<td>test listing files in directory.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/04-file-stat.p.html">04-file-stat.p</a></td>
<td>check properties of a file</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/04-multi-assign.p.html">04-multi-assign.p</a></td>
<td>test multi value assignment</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/04-strace-test.p.html">04-strace-test.p</a></td>
<td>show stack trace of deep nested function (showstack function)</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-time.p.html">04-time.p</a></td>
<td>get current date / time; functions: localtime, gmtime, currenttime</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/06-func-obj-call.p.html">06-func-obj-call.p</a></td>
<td>function invokes argument callback over range of numbers; prints result as ASCII graph</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/08-object-prototype-finalizer.p.html">08-object-prototype-finalizer.p</a></td>
<td>test where object has a finalizer - it is called when object is freed.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/08-object-prototype-point.p.html">08-object-prototype-point.p</a></td>
<td>0bject oriented programming with prototypes: point class</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/08-object-prototype-stack.p.html">08-object-prototype-stack.p</a></td>
<td>0bject oriented programming with prototypes: stack class</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/10-object-closure-point.p.html">10-object-closure-point.p</a></td>
<td>Object oriented programming with closures: point class</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/10-object-closure-stack.p.html">10-object-closure-stack.p</a></td>
<td>Object oriented programming with closures: stack class.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/11-file-pipe.p.html">11-file-pipe.p</a></td>
<td>run process and script send data to it via stdin and read stdout</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/11-file-test.p.html">11-file-test.p</a></td>
<td>working with files</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/15-thread-range.p.html">15-thread-range.p</a></td>
<td>co-routine threads: create thread via API, thread yields a range of numbers.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/15-thread-range-1.p.html">15-thread-range-1.p</a></td>
<td>create co-routine thread via api</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/15-thread-range-fib.p.html">15-thread-range-fib.p</a></td>
<td>co-routine threads: create thread via API, thread yields sequence of fibonaci numbers.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/16-load-extension-library.p.html">16-load-extension-library.p</a></td>
<td>load an extension library written in C and call exported function</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/17-for-lib.p.html">17-for-lib.p</a></td>
<td>co-routines / generators as used with for statement</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/17-for-range.p.html">17-for-range.p</a></td>
<td>co-routines / generators as used with for statement</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/17-for-range-lib.p.html">17-for-range-lib.p</a></td>
<td> </td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/17-for-while-break-skip.p.html">17-for-while-break-skip.p</a></td>
<td>for loop ; test keywords: break - finishing the iteration and skip - continue to next iteration.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/18-higher-order-func.p.html">18-higher-order-func.p</a></td>
<td>Higher order functions fold/filter/map implemented as script functions.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/18-higher-order-func-lib.p.html">18-higher-order-func-lib.p</a></td>
<td>Higher order functions fold/filter/map - use built in library implementation.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/18-higher-order-func-lib2.p.html">18-higher-order-func-lib2.p</a></td>
<td>Higher order functions fold/filter/map - use built in library implementation.</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/18-sort.p.html">18-sort.p</a></td>
<td>use standard library sort procedure - with callback for comparison</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/18-sort-anonymous.p.html">18-sort-anonymous.p</a></td>
<td>use standard library sort procedure - with anonymous function as callback for comparison</td>
</tr>
</tbody>
</table>
<p>The next table lists some libraries that can be included - all written in pooh</p>
<table>
<thead>
<tr>
<th> </th>
<th> </th>
</tr>
</thead>
<tbody>
<tr>
<td><em>Test name</em></td>
<td><em>Description</em></td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/01-complex.p.html">01-complex.p</a></td>
<td>test complex number library - complex.inc</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/01-matrix.p.html">01-matrix.p</a></td>
<td>test matrix library - matrix.inc</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/02-heap.p.html">02-heap.p</a></td>
<td>test heap data structure - heap.inc</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/02-heap2.p.html">02-heap2.p</a></td>
<td>test heap data structure - heap.inc</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-queue.p.html">03-queue.p</a></td>
<td>test queue data structure - queue.inc</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/03-stack.p.html">03-stack.p</a></td>
<td>test stack data structure - stack.inc</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/04-slist.p.html">04-slist.p</a></td>
<td>test linked list data structure - slist.inc</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/05-graphlist.p.html">05-graphlist.p</a></td>
<td>test adjacency graph data structure - graphlist.inc</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/05-graphmatrix.p.html">05-graphmatrix.p</a></td>
<td>test adjacency matrix data structure - graphmatrix.inc</td>
</tr>
<tr>
<td><a href="/cstuff/docs/html/pooh/example/06-graphutils.p.html">06-graphutils.p</a></td>
<td>test graph traversal class - graphutils.inc</td>
</tr>
</tbody>
</table>
<h3 id="so-what-is-next-">So what is next ?</h3>
<p>So far I did not test the whole concept with real clients - nobody has ever used the Pooh language in order try to teach how to program. Well, it has to start with someone ?.</p>
<p>The concept of passing all function parameters as named parameters maybe a consistent and readable choice, but it is probably more difficult to write the program text this way. What may help is the addition of a special editing program - an IDE (Interactive developing environment) like Eclipse but with a user interface adapted for young people. Here the program parameter names will pop up as hints, together with lots of explanations about the function and its parameters. Now that is a lot of work.</p>
<p>Another important direction is addition of practical parsing engine ; similar to those of REBOL or Perl 6.</p>
<p>In the meantime I will continue to</p>
<ul>
<li>Test the system under different configurations</li>
<li>Add missing library functions;</li>
<li>Complete the introduction text.</li>
</ul>
<p>So maybe all we have is an eclectic set of features chosen by virtue of my personal preferences and not because of any objective and measurably better usability. Maybe that’s the way programming languages are progressing; it is hard to compare approaches so that one can make a valid generalization; no tool is going to be the best in all cases, and anyway comparison are often between apples to oranges. Well enough quetching already, lets get some work done!</p>
<h3 id="and-now-for-something-completely-different">And now for something completely different</h3>
<p>The best pooh movies are here</p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/sqdiEUp6s4E" frameborder="0" allowfullscreen=""></iframe>
<iframe width="420" height="315" src="http://www.youtube.com/embed/qam9JBk5Oig" frameborder="0" allowfullscreen=""></iframe>
Pooh, the programming language
2012-12-11T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/12/11/post-51
["", "blogg"] | <notextile>Pooh, the programming language</notextile>
<hr />
<p class="publish_date">
11 December 2012
</p>
<p><br /><br />I just released the pooh programming language for kids and grown ups. <a class="ot-anchor" href="http://mosermichael.github.com/cstuff/all/pooh-lan/2012/12/11/pooh-intro.html" rel="nofollow">http://mosermichael.github.com/cstuff/all/pooh-lan/2012/12/11/pooh-intro.html</a> Please take a look; It is hanukkah and this is my candle light ;-) a Happy Hanukkah for everyone who celebrates.</p>
curious butterflies
2012-10-21T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/10/21/post-13
["", "blogg"] | <notextile>curious butterflies</notextile>
<hr />
<p class="publish_date">
21 October 2012
</p>
<p><br /><br />These butterflies migrate between UK and Africa, they reach an altitude of 1000 meters, and one direction takes six generations of butterflies. Now where do them butterflies ask for directions ?<br /><a class="ot-anchor" href="http://www.bbc.co.uk/nature/19991550" rel="nofollow">http://www.bbc.co.uk/nature/19991550</a><br /><br />Also amazing: the metabolism of the hummingbird<br /><a class="ot-anchor" href="https://en.wikipedia.org/wiki/Hummingbird#Metabolism" rel="nofollow">https://en.wikipedia.org/wiki/Hummingbird#Metabolism</a><br />the tiny heart of a hummingbird can beat up to 1,260 beats per minute; and the bird flies distances of 800km, to conserve energy, it enters hibernation at night.</p>
don't rock the boat
2012-10-09T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/10/09/post-18
["", "blogg"] | <notextile>don't rock the boat</notextile>
<hr />
<p class="publish_date">
09 October 2012
</p>
<p>Wow! The SpaceX rocket has nine engines; yesterday it was launched, one of them engines explodes in a big puff but the ship kept going steady</p>
<p><a href="http://arstechnica.com/science/2012/10/that-smooth-spacex-launch-turns-out-one-of-the-engines-exploded/">article</a></p>
<p>i gess the Best song about launching stuff into space is this one:</p>
<p><a href="https://www.youtube.com/watch?v=4I3wFEfWeqo">Link</a></p>
<iframe width="420" height="315" src="//www.youtube.com/embed/4I3wFEfWeqo" frameborder="0" allowfullscreen="">
</iframe>
quote of the day
2012-09-27T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/09/27/post-54
["", "blogg"] | <notextile>quote of the day</notextile>
<hr />
<p class="publish_date">
27 September 2012
</p>
<p><br /><br />"Life was simple before World War II. After that, we had systems." Grace Hopper <a class="ot-anchor" href="http://en.wikiquote.org/wiki/Grace_Hopper" rel="nofollow">http://en.wikiquote.org/wiki/Grace_Hopper</a></p>
<p>Other quotes from the interent, I added them to the bugzilla quips at work:</p>
<pre>
C++ : increment the value of C and use the old one
"Everyone knows that debugging is twice as hard as writing a program in the first place.
So if you're as clever as you can be when you write it, how will you ever debug it?"
Brian W. Kernighan and P. J. Plauger ("The Elements of Programming style")
"When in doubt, use brute force" Ken Thompson
Simplicity is prerequisite for reliability. (Edsger Dijkstra)
Programs for sale: fast, reliable, cheap - choose two. ~Author Unknown
"Odd that we think definitions are definitive." Larry Wall
"In times of crisis, it is of utmost importance not to lose one's head." -- M. Antoinette
"Computers in the future may weigh no more than 1.5 tons." - Popular Mechanics, 1949
A good advice from Bugs Bunny: Jump to the next bug and be happy.
There are two ways to write error-free programs; only the third one works.
The proof of a system's value is its existence.
... and then I went from "why doesn't this work?" to "why did this ever work?"
\[expletive deleted\]
Can we fix it? Yes, we can!
War is Peace. Freedom is Slavery. Ignorance is Strength.
Parse error at byte 9 (line 1): unexpected token
Programmer: an organism that turns coffee into software
One man's crappy software is another man's full time job. ~Jessica Gaston
“Quotes on the internet are often inaccurate” - Abraham Lincoln
</pre>
politics of product design
2012-09-05T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/09/05/post-50
["", "blogg"] | <notextile>politics of product design</notextile>
<hr />
<p class="publish_date">
05 September 2012
</p>
<p><br /><br />The Vasa was a battle ship that sank on its maiden voyage - sort of a symbol for bad design .<br />… it was all politics "Swedish king Gustavus Adolphus, who was leading the army on the continent on the date of its maiden voyage, was impatient to see it join the Baltic fleet in the Thirty Years' War; at the same time, the king's subordinates lacked the political courage to discuss the ship's structural problems frankly or to have the maiden voyage postponed. An inquiry was organized by the Swedish privy council to find personal responsibility for the disaster, but in the end no one was punished for the fiasco. "<br /><br /><a class="ot-anchor" href="http://en.wikipedia.org/wiki/Vasa_(ship)" rel="nofollow">http://en.wikipedia.org/wiki/Vasa_(ship)</a><br /><br />I think the subject matter would make a great play, epic theater production, controversial and always up to date ;-)</p>
Mind expansion
2012-08-22T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/08/22/post-39
["", "blogg"] | <notextile>Mind expansion</notextile>
<hr />
<p class="publish_date">
22 August 2012
</p>
<p><br /><br />A great comment…<br /><br />From here: <a class="ot-anchor" href="http://lwn.net/Articles/508865/#Comments" rel="nofollow">http://lwn.net/Articles/508865/#Comments</a></p>
<pre>
> and the speed of light, which has remained stubbornly constant.
Anyone has a patch for this? ;-)
</pre>
Food prices and the larger scheme of things
2012-08-09T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/08/09/post-1
["", "blogg"] | <notextile>Food prices and the larger scheme of things</notextile>
<hr />
<p class="publish_date">
9 August 2012
</p>
<p>Food prices are <a href="http://www.bbc.co.uk/news/business-19193390">about to rise</a> sharply; Now for me steep price changes create a feeling of general insecurity, I guess that I am not alone here;
a general feeling of insecurity is then followed by less spending in general; and that follows straight into a recession; (How do you quantify this reasoning?)</p>
<hr />
<p>My alternative history of the economic crisis that started in 2007</p>
<p>Before 2008 there was a strong assumption that the inflation and interest rates would be low; this notion was enforced by the fact that inflation did not rise despite a <a href="http://en.wikipedia.org/wiki/File:Crude_oil_prices_since_1861_(log).png">sharp growth</a> in oil/energy prices throughout the decade;
This trick was achieved by redefining/tricking the way that economic indicators (CPI/PCE) <a href="http://en.wikipedia.org/wiki/United_States_Consumer_Price_Index#Perceived_underestimation_of_inflation">where calculated</a></p>
<p>I guess these assumptions were the reason why <a href="http://en.wikipedia.org/wiki/Subprime_mortgage_crisis">subprime mortgages</a> where valuated as AAA (really valuable assets) ; with low inflation numbers the mortgage rates would be low forever; what could possibly go wrong ? And so the whole economy was a house of cards that was based on massaging of the statistics that defined the rate of inflation.</p>
<p>Now the cards came down because of the <a href="http://en.wikipedia.org/wiki/2007%E2%80%9308_world_food_price_crisis">world food price crisis of 2007</a> . I did not find any explanation on the web as to why inflation started to grow during 2007; but lets remember that it coincided with the sharp rise in food prices. I am not sure how, but it seems that somehow it was no longer possible to massage the inflation numbers while basic foods prices went through the roof.</p>
<p>The rest is history: people did not meet the mortgage payments due to rising interest rates; these of course are linked to inflation numbers; the banks went down, the economy freezes.</p>
<hr />
<p>Now food prices are related to the price of oil; the weather; all sorts of reason beyond our control; Very humbling for us.</p>
<p>Like it said in the Shema Yisrael prayer ‘ .. And it will come to pass that if you continually hearken to My mitzvot that I command you today, to love the Lord your God, and to serve Him, with all your heart and with all your soul ? then I will provide rain for your land in its proper time, the early and late rains, that you may gather in your grain, your wine, and your oil. I will provide grass in your field for your cattle and you will eat and be satisfied.’ http://www.aish.com/jl/m/pb/48954656.html</p>
<hr />
news in TCP/IP land
2012-08-08T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/08/08/post-40
["", "blogg"] | <notextile>news in TCP/IP land</notextile>
<hr />
<p class="publish_date">
08 August 2012
</p>
<p><br /><br />Explains the new tcp option TCP_FASTOPEN that will speed up tcp connection by allowing sending of data with first SYN packet (but only for the second, third and so on connections to a single host).<br /><br /><a class="ot-anchor" href="http://lwn.net/SubscriberLink/508865/0785b746321ef4f3/" rel="nofollow">http://lwn.net/SubscriberLink/508865/0785b746321ef4f3/</a><br /><br />Interesting.</p>
Forgotten anniversary
2012-07-30T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/07/30/post-21
["", "blogg"] | <notextile>Forgotten anniversary</notextile>
<hr />
<p class="publish_date">
30 July 2012
</p>
<p><br /><br />75 years ago, the orders that started the Great Terror were signed. It's a day to remember the victims of Stalinism.<br /><br /><a class="ot-anchor" href="http://en.wikipedia.org/wiki/NKVD_Order_No._00447" rel="nofollow">http://en.wikipedia.org/wiki/NKVD_Order_No._00447</a></p>
e-book readers call home
2012-07-05T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/07/05/post-19
["", "blogg"] | <notextile>e-book readers call home</notextile>
<hr />
<p class="publish_date">
05 July 2012
</p>
<p><br /><br />I am deeply shocked; e-book readers call back home, so they know what and how you are reading stuff. E-book readers as instruments of surveillance :-( <a class="ot-anchor" href="http://online.wsj.com/article/SB10001424052702304870304577490950051438304.html" rel="nofollow">http://online.wsj.com/article/SB10001424052702304870304577490950051438304.html</a><br /><br />The PC was once an open tool; so they came up with all those closed gadgets that are supposed to replace the PC; And the irony is that all of them are running Linux or BSD ;-)<br /><br />… somehow GPLv3 starts to make sense now; an I thought it was too restrictive when it came out ;-(</p>
Jewish matters ignored in Israel
2012-06-28T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/06/28/post-35
["", "blogg"] | <notextile>Jewish matters ignored in Israel</notextile>
<hr />
<p class="publish_date">
28 June 2012
</p>
<p><br /><br />A court in Germany outlaws circumcision; the press in Israel does not cover the case. I think they should fire all journalists here.<br /><br /><a class="ot-anchor" href="http://www.faz.net/aktuell/politik/inland/urteil-zur-beschneidung-von-jungen-schnitt-und-schmerz-11801177.html" rel="nofollow">http://www.faz.net/aktuell/politik/inland/urteil-zur-beschneidung-von-jungen-schnitt-und-schmerz-11801177.html</a></p>
In China hinter der Mauer
2012-06-20T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/06/20/post-31
["", "blogg"] | <notextile>In China hinter der Mauer</notextile>
<hr />
<p class="publish_date">
20 June 2012
</p>
<p><br /><br />I think Mr. Bao Tong has a face that is both essentially chearful and very wise and cynical. Maybe he is Jewish ? <a class="ot-anchor" href="http://www.nybooks.com/blogs/nyrblog/2012/jun/14/china-corruption-bao-tong-interview/" rel="nofollow">http://www.nybooks.com/blogs/nyrblog/2012/jun/14/china-corruption-bao-tong-interview/</a><br /><br />One guy who had to get out of Czechoslovakia after 1967 told me that his moment of enlightenment was when both Chinese and Czech were laughing over the same political jokes<br /><br />Talking about boat rocking, what about the bonus system in banking / sky high CEO compensation ? In other words there is always a place for cronies in any political system.<br /><br /><a class="ot-anchor" href="http://www.youtube.com/watch?v=4JSO5dAa_bk" rel="nofollow">Wolf Biermann - In China hinter der Mauer</a></p>
"россия велика, всю не разворуеш" (TM)
2012-06-18T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/06/18/post-64
["", "blogg"] | <notextile>"россия велика, всю не разворуеш" (TM)</notextile>
<hr />
<p class="publish_date">
18 June 2012
</p>
<p><br /><br />phrase is original, it has not been found on any search engines.</p>
EUSSR
2012-06-10T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/06/10/post-20
["", "blogg"] | <notextile>EUSSR </notextile>
<hr />
<p class="publish_date">
10 June 2012
</p>
<p>A very interesting readers opinion in the FAZ (Frankfurter Algemeine Zeitung); Here is my translation:</p>
<p>By WINFRIED BÖHME</p>
<p>“There is no convincing argument in favor of a centralized European state. One can discern the following two arguments for a centralized Europe: 1) The centralized state will bring about a permanent state of peace 2) The relative weight of Germany is quite low, while with a centralized Europe we would have a greater relative weight in the world.
The first argument is plain rubbish: it is refuted by European history in the post war period. The second argument can’t be easily discarded: The USSR was a superpower because of unity, the USA are still a superpower because of unity. Now we should be asking the following question: why exactly do we need a European superpower? Against whom? What is the price of for this aspiration of powers ? The price seems to be clear: it is the destruction of European pluralism, its bounty of different mentalities and its plentitude of cultures by means of an imposition of discipline and order. It implies paternalism, control and an imposed uniformity;
In other words the price to be is the theft of liberty and individuality for each of the nations of Europe.”</p>
<p>Source: <a class="ot-anchor" href="http://www.faz.net/aktuell/wirtschaft/schuldenkrise-zerbricht-jetzt-der-euro-11779705.html#comments" rel="nofollow">http://www.faz.net/aktuell/wirtschaft/schuldenkrise-zerbricht-jetzt-der-euro-11779705.html#comments</a></p>
<p>Lesermeinung in der FAZ, von WINFRIED BÖHME</p>
<p>Es gibt keine wirklich überzeugende Idee für einen europäischen Zentralstaat!
Als Begründung für ein zentralistisches Europa hört man regelmäßig zwei Gründe: 1. Der Einheitsstaat Europa wäre eine Stabilisierung des Friedens. 2. Das Gewicht von Deutschland wäre so gering, mit der europäischen Einigung gäbe es mehr Gewicht in der Welt.
Das erste Argument ist schlichtweg Unfug, historisch widerlegt durch die Nachkriegsgeschichte. Das zweite Argument ist nicht so leicht von der Hand zu weisen: Die UdSSR war eine Supermacht durch Zusammenschluss, die USA sind es noch. Hierzu wäre genauer zu fragen, warum sollen wir eine europäische Supermacht gründen? Gegen wen eigentlich? Und was wäre der Preis für dieses Machtstreben? Dieser Preis scheint inzwischen absehbar: Er wäre die Zerstörung des europäischen Pluralismus, seinen Reichtum an Mentalitäten und die Vielfalt seiner Kulturen durch „Zucht und Ordnung“, also Bevormundung, Kontrolle und Vereinheitlichung. Mit anderen Worten, ein Raub an Freiheit und Individualität für jede Nation innerhalb dieses Europas.</p>
<p>Lesermeinung zum Artikel <a class="ot-anchor" href="http://www.faz.net/aktuell/wirtschaft/schuldenkrise-zerbricht-jetzt-der-euro-11779705.html#comments" rel="nofollow">http://www.faz.net/aktuell/wirtschaft/schuldenkrise-zerbricht-jetzt-der-euro-11779705.html#comments</a> (in der Faz sind die Lesermeinungen intelligenter als es die Artikel selbst sind)</p>
<hr />
<p>However there is always a different interpretation: the single market has removed barriers of entry for middle range firms; instead of having to deal with export regulations they can focus on the trade, as if it were a domestic market. That puts a lot of agility into the hands of medium sized firms; now everything that doesn’t favor big players in the economy exclusively is a very equalizing element that works against established monopolies. Everything that is bad for the big players is good for demoracy, sort of. Now go figure out who is right and who is wrong.</p>
Bradbury is dead .
2012-06-07T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/06/07/post-8
["", "blogg"] | <notextile>Bradbury is dead .</notextile>
<hr />
<p class="publish_date">
07 June 2012
</p>
<p><br /><br />
RIP <a href="https://en.wikipedia.org/wiki/Ray_Bradbury">Ray Bradbury</a></p>
<p>And you can fly<br />High as a kite if you want to<br />Faster than light if you want to<br />Speeding through the universe<br />Thinking is the best way to travel<br /><a class="ot-anchor" href="http://www.youtube.com/watch?v=36lJMpzbOA4">The Moody Blues - The Best Way To Travel</a></p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/36lJMpzbOA4" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>Back in the eighties, as a teengager, i spent a lot of time learning how to program that TI99/4A home computer. Now I think it was sci-fi that got me motivated: that home computer was the stuff made right out of the dreams of science fiction!
Thanks Ray for getting me going…</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/_W-r7ABrMYU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>Here is the man talking.</p>
Lecture on analogy and cognition
2012-05-09T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/05/09/post-36
["", "blogg"] | <notextile>Lecture on analogy and cognition</notextile>
<hr />
<p class="publish_date">
09 May 2012
</p>
<p><br /><br />Lecture by Douglas Hofstadter ; author of Gödel, Escher, Bach. <a class="ot-anchor" href="http://www.youtube.com/watch?v=n8m7lFQ3njk" rel="nofollow">Analogy as the Core of Cognition</a></p>
it can be traced
2012-03-22T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2012/03/22/post-33
["", "blogg"] | <notextile>it can be traced</notextile>
<hr />
<p class="publish_date">
22 March 2012
</p>
<p><br /><br />Very interesting PERL module; Tracing is better than debugging; Bash has set -x - PERL can do this too.<br /><br /><a class="ot-anchor" href="http://search.cpan.org/~mob/Devel-DumpTrace-0.15/lib/Devel/DumpTrace.pm" rel="nofollow">http://search.cpan.org/~mob/Devel-DumpTrace-0.15/lib/Devel/DumpTrace.pm</a></p>
great vim extension
2011-12-19T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2011/12/19/post-26
["", "blogg"] | <notextile>great vim extension</notextile>
<hr />
<p class="publish_date">
19 December 2011
</p>
<p><br /><br />Life changing VIM extension - does autocompletion based on ctags database <a class="ot-anchor" href="http://www.vim.org/scripts/script.php?script_id=1520" rel="nofollow">http://www.vim.org/scripts/script.php?script_id=1520</a></p>
other stars
2011-12-05T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2011/12/05/post-47
["", "blogg"] | <notextile>other stars</notextile>
<hr />
<p class="publish_date">
05 December 2011
</p>
<p><a href="http://www.nasa.gov/mission_pages/kepler/news/kepscicon-briefing.html">NASA’s Kepler Mission Confirms Its First Planet in Habitable Zone of Sun-like Star</a></p>
<p>very exciting news!</p>
Steve Jobs the tweaker
2011-11-13T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2011/11/13/post-56
["", "blogg"] | <notextile>Steve Jobs the tweaker</notextile>
<hr />
<p class="publish_date">
13 November 2011
</p>
<p><a href="http://www.newyorker.com/reporting/2011/11/14/111114fa_fact_gladwell?currentPage=all">Malcolm Gladwell on Steve Jobs</a> There are inventors, and there are tweakers - these make small but important adaptations that make the big thing work; Malcolm Gladwell says that Jobs was a tweaker. Would that make him an Englishmen?</p>
<p>I still think the this is not quite correct; Steve Jobs was responsible for several innovations, the adoption of <a href="http://en.wikipedia.org/wiki/Graphical_user_interface">graphic user interfaces</a> on mainstream computers and the invention of <a href="http://en.wikipedia.org/wiki/TrueType">true type fonts</a> . Graphic user interfaces were invented by the <a href="http://en.wikipedia.org/wiki/PARC_(company)">Xerox PARC</a> , the concept worked on workstation computers that cost more than 10000$ and Xerox did not see the idea as something interesting. Jobs made it to work on the modest personal computers of the 1980ies, and this was quite a huge leap, it was something barely possible. For this he earns my respect.</p>
<p>Of course Jobs also brought about the shift in computing away from the relatively open PC platform to closed gadgets, like the iPhone, where the end user does not have full control over the device. This move has some internal logic - in the PC world there are lots of players, one company is producing the computer, another one the operating system etc. etc. here every player gets a piece of the pie. However the profit margin for a single player are higher if this single player can sell a larger and more integrated package.</p>
<p>The limited pie was good enough, when the PC market was growing fast; this is no longer so, therefore the big vendors try to increase their own margin by selling integrated systems. This trend was not created by just one person, so it goes.</p>
<hr />
<p>i read parts of Steve Jobs by Walter Isaacson; what strikes me is the super sized ego of Jobs; at the same time he is said to have dropped acid and he was into Buddhist religion. Somehow the concept of Ego death and super sized ego do not seem to match very well.</p>
<p>Ego death would be a temporarily suspension of the ego (i guess that a permanently suspended ego would be a sign of psychological damage). The Ego is like a shield of the self: all incoming information is evaluated by means of the ego - this results in strong filtering, for instance one is shielded/unable to perceive inconvenient truths.</p>
<p>temporary suspension of ego may result in a new kind of openness; for the information filter may no longer be absolute. For some positions the gain of new insights may out-weight associated negative effects.</p>
<p>Maybe the right balance way would be for the ego to be both there and not there; don’t know if that is possible and don’t know if Jobs has reached this stage though.</p>
Olden times
2011-10-31T00:00:00+00:00
http://mosermichael.github.com//all/blogg/2011/10/31/post-44
["", "blogg"] | <notextile>Olden times</notextile>
<hr />
<p class="publish_date">
31 October 2011
</p>
<p><br /><br />Life turned sour, when they invented agriculture; Too much work, new diseases, rulers who turned into owners … ; Interesting that you can read it in the Bible: Cain, the crop farmer is the evil one, jealous of his brother Abel, the shepherd. And Cains offerings suck too;<br /><br /><a class="ot-anchor" href="http://zompist.wordpress.com/2011/10/29/why-you-want-to-be-a-hunter-gatherer/" rel="nofollow">http://zompist.wordpress.com/2011/10/29/why-you-want-to-be-a-hunter-gatherer/</a></p>
So what about the tides hat shape history?
2011-10-27T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2011/10/27/forces-of-history
["", "ramblings"] | <h2 id="so-what-about-the-tides-hat-shape-history">So what about the tides hat shape history?</h2>
<p class="publish_date">
18 July 2011
</p>
<p>They say there are longer term business cycles, called <a href="http://en.wikipedia.org/wiki/Kondratiev_wave">Kondratiev waves</a> , Interesting idea.</p>
<p>Well, I don’t know much about business, there seem to be longer term cycles in politics too. From the 1940’s to the 1970’s there has been a period defined by major commonalities in the policies of the leading countries in the Western world; this approach has different names in different countries: New Deal in the US / Social democratic consensus in GB / Soziale Marktwirtschaft in DE (yes, and I don’t know French ;-).</p>
<p>Whatever it’s name, the policy ended with a ten year economic malaise that was to a large part brought about by these same policies. I remember this little book <a href="http://www.amazon.com/Capitalism-Beginners-Robert-Lekachman/dp/0394738632">Capitalism for beginners</a> where the conflicts where explained in great depth and beauty as a an illustrated comic book.</p>
<p>From the 1980s to 00’s, for roughly thirty years, there has been another period of common approaches; this started by a the congruential policies of Ronald Reagan and Margaret Thatcher; again it is known as Reaganomics / Thatcherism / <a href="http://en.wikipedia.org/wiki/Laissez-faire">Laissez-fair capitalism</a> . Whatever it all ended in a large bust, also brought about by the effect of same policies; Deregulated banks and rating agencies brought it about; globalization/global reach of finance created vast financial empires that are now ‘too big to fail’, whatever these non-state entities do, it all affects us profoundly; Please note also a marked shift of focus from the nation state to non-state financial empires and super national constructs such as the European Union, all this becomes apparent as the crisis unfolds.</p>
<p>It’s not exact to say that globalization created financial institutions that are ‘too big to fail’; More correct is that the liberated financial institutions enabled an expansion of businesses into foreign domains; it also financed the internet bubble of the late 90ies, which as a valuable by-product created a global communication network, the internet, this alone was an invaluable aid to globalization. Expanded businesses were then feeding back the banks; So this feedback loop created huge financial empires.</p>
<p>Globalization and huge banks go together; huge trade operations have to be insured, well any transaction has to be insured in some way, (See <a href="http://en.wikipedia.org/wiki/Trade_credit_insurance">Trade Credit Insurance</a> ) So huge insurance agencies are created and are backed by even larger banks.</p>
<h3 id="what-will-come-next">What will come next?</h3>
<p>Should one now expect a major paradigm shift that would have a lasting effect for the coming 30 year period ? Next time it can’t be a return to previous social democratic system, for the major economies have accumulated vast amounts of debt and cannot tolerate large new expenditures. It can’t be just a continuation of the previous laissez-fair approaches, globalization/global reach of finance created vast financial empires that are now ‘too big to fail’, all large scale operations do screw up occasionally; of course screw ups can happen either by accident or by design. In either event these huge financial empires would have to be bailed out periodically, as it stands now, society will not tolerate periodic rescue missions that entail nationalization of losses, while expecting profits to remain privatized.</p>
<p>What comes now now are speculations on possible policy changes, what directions can be considered.</p>
<p>Is there a possibility to scale back/downsize globalization? After all, without it there could be no huge financial empires, those empires would have to forgo a process of shrinking back. It would be difficult for nations to re-impose trade barriers though; the modern production process is probably too complicated for a medium sized nation state to handle on its own (Well with the exception of North Korea); However both US and EU may tighten up it’s boarders and China can turn inwards. Could such a major change be brought about without violent conflicts? The last downsizing of globalization occurred as a result of World War I.</p>
<p>Maybe change will be brought about by the players themselves; For example China might seek to divest from US sovereign debt for example, the technocrats in charge of China might seek to compartmentalize operations, a huge ship divided into compartments runs a lower risk of sinking; A modern downsizing of globalization might also bring about a fragmentation of the Internet; (i would hate that); the dismantling of the global internet may also come about as a result of cyber weapons, sort of a preemptive defense against stuxnet and its friends.</p>
<p>One tiny model for de-globalization might be the story of Iceland; Before the recession, Iceland was a banking sector that had a nation; the banks where much larger than the national economy; now three years later the economy of Iceland is doing fine; unlike other nations, they did not bail out the banks; they just couldn’t do that, given that the banks where much larger than the national economy, this act evoked the wrath of other nations; Anti-terrorism laws where <a href="http://www.bloomberg.com/apps/news?pid=newsarchive&sid=aXjIA5NzyM5c">invoked to seize Icelandic bank assets in GB</a> . This very hostile <a href="http://www.bbc.co.uk/iplayer/episode/b015v1q6/HARDtalk_Steingrimur_Sigfusson_Icelands_Finance_Minister/">BBC interview on HARDtalk</a> tries to rip apart the valiant Icelandic finance minister Steingrimur Sigfusson</p>
<p>However a major economic decoupling between the power blocks runs the increased risk of political tensions and conflict between the large power blocks. What about stability during the last global standoff, the Cold War? Depending on the assessment of this issue one will see this as a good/bad/terrible scenario.</p>
<p>So we might stay with globalization, all with the aim to maintain Peace on Earth / Peace between the power blocks. Maybe financial empires will not be touched, and they will still require periodical bailouts of epic proportions; this of course runs the risk of disturbing internal peace. So this analysis exposes an interesting conflict between maintaining an intertwined global economy on the one hand and maintaining internal stability / social peace on the other hand. One possible approach is to scale back the standard of living in the western world, all in order to get the means of financing the big project; this can be justified by the need to care for the environment; after all all the planet can’t sustain it all, can’t it? Of course a strengthening of the police and security forces brought about by the war on terror also helps with maintaining the peace; Under this scenario the world would look more like the Austro-Hungarian empire before the Great War; conservative politics and a lots of internal tensions would create an atmosphere of a kettle that is about to burst; so maybe ubiquitous surveillance with CCTV cameras is the message for the next thirty years.</p>
<p>The ‘soul searching’ for a new modus vivendi usually lasts for a decade; so it is still too early to say which way will be chosen; maybe even a combination of both; For example the Euro fails, then there will be local down scaling within Europe, while the US may choose to muddle along (if it stays out of the falling debris).</p>
<h3 id="further-reasons-for-hope-and-despair">Further reasons for hope and despair</h3>
<p>However, there might come technological innovations that bring about a change in the equations; In a larger sense this seems to be strongest argument of the optimists.</p>
<p>Of course Capitalism has been the system that encourages innovation in the strongest form; this system has hosted the most number of innovations and thankfully, this system has therefore prevailed so far.</p>
<p>Now during the 70ies some analyst where projecting a process of convergence between the systems of modern Capitalism on the one hand, and State Capitalism of the USSR on the other hand; if these guys do have a point then such a convergence would pose a considerable threat to the continued success of Capitalism as such. A requirement for periodical bail outs can be seen as a move towards State Capitalism; State Capitalism is a very coercive system; every initiative has to originate strictly from top top down; and as such this system does not favor innovation too much.</p>
<p>I hope it will not get that far; there is still too much competition inherent in the system; If one takes for example the high-tech sector; then this industry has seen several waves of consolidation; each period has had its succession of champions - IBM/Microsoft/Google; on each occasion no player was strong enough to dominate the industry completely. Maybe this was due to external pressure though; in the 70ies and early 80ies IBM was subject to a serious anti-trust challenge, so maybe fear motivated them not to deal with PC software too much, all on purpose; in the late 90ies Microsoft was subject to a serious anti-trust investigation, so maybe they did not bother with search engines / online advertising / online retail - again on purpose; very hard to say what is real here and what is not so real, after all its all just software; empires in the sand.</p>
HTTP server with co-routines
2011-10-25T00:00:00+00:00
http://mosermichael.github.com//all/projects/2011/10/25/http-server-with-coroutines
["", "projects"] | <h2 id="http-server-with-co-routines">HTTP server with co-routines</h2>
<p class="publish_date">
25 October 2011
</p>
<p>The <a href="http://mosermichael.github.com/cstuff/all/projects/2011/10/09/Non-Blocking-IO-with-coroutines.html">previous article</a> describes a co-routine library integrated with the <a href="http://libevent.org">libevent</a> asynchronous event notification library, this article now presents WEBBY - an HTTP server that comes in two implementations, one that uses the co-routine library, and another one that handles each connection in a different operating system thread. WEBBY is an <a href="http://en.wikipedia.org/wiki/Embedded_HTTP_server">embedded HTTP server</a> , that means it should be easy to integrate the server with an existing application, and its main focus is on creating dynamic WEB content.</p>
<p>There are many tools that promise to do non blocking network communication properly; most of them come with a HTTP server. There are many reasons for this, first of all doing a WEB server is a means to test the tool. The author of a tool is then the first one to discover shortcomings of said tool, the proper computer jargon for this process is <a href="http://en.wikipedia.org/wiki/Eating_your_own_dog_food">eating ones own dog food</a> , and so we have a valid usage of ‘dog-food’ as a noun, alas it is limited to the practice of programming a computer.</p>
<p>Also of course HTTP is the ‘little protocol that could’, nowadays all sorts of application are built on top of HTTP; web services, video and audio streaming, of course it serves this web page as well; so by implementing an HTTP server one states that the tool is at least good for serving web pages;</p>
<h3 id="the-many-functions-of-a-web-server-and-its-design-by-filtersmodules">The many functions of a WEB server and its design by filters/modules.</h3>
<p>The HTTP protocol consists of a set of non overlapping features that complement each other without interfering with each other, this state is called <a href="http://en.wikipedia.org/wiki/Orthogonality#Computer_science">Orthogonal set of features</a> . For example the protocol has the option of <a href="http://en.wikipedia.org/wiki/HTTP_compression">compressing the request/response body</a> , which does not bother the feature of <a href="http://en.wikipedia.org/wiki/Basic_access_authentication">HTTP Basic Access authentication</a></p>
<p>The system is therefore implemented as a <a href="http://en.wikipedia.org/wiki/Pipeline_(computing)">pipeline of handlers</a> where each feature is implemented by a dedicated handler, the output of one handler is automatically used as the input to the next, following handler (very common theme with programming ;-). Each such handler is called a filter, all usable HTTP servers have this concept (<a href="http://httpd.apache.org">Apache</a> and <a href="http://nginx.org/">nginx</a> it
is called a ‘module’)</p>
<p>Each filter must be able to alter a particular aspect of the HTTP request or response, so each filter will have a chance to alter one of the following events</p>
<ul>
<li>The event that a <a href="http://mosermichael.github.com/cstuff/docs/html/webby/a00019.html#ga27407d3eb9a137407a8432603325b5b0">http header has been parsed</a> ; an object representing the parsed <a href="http://mosermichael.github.com/cstuff/docs/html/hutils/a00024.html">HTTP request</a> header in memory</li>
<li>The event that <a href="http://mosermichael.github.com/cstuff/docs/html/webby/a00019.html#ga07fb25acde5fb71c0bbcad748331e462">http request data has been parsed</a> ; zero or more blocks of data that make up the HTTP request body (without <a href="http://en.wikipedia.org/wiki/Chunked_transfer_encoding">chunk headers</a>)</li>
<li>An event that signals that <a href="http://mosermichael.github.com/cstuff/docs/html/webby/a00019.html#gaf6f5ee2b34c30cd71ddeccf8a8cc98c8">HTTP request has been parsed completely</a></li>
</ul>
<p>These events make up one direction, the processing of the request, at some stage a filter will generate a response, which will traverse the same sequence of filters backwards. Now each filter will have a chance to alter one of the following events</p>
<ul>
<li>The event that a <a href="http://mosermichael.github.com/cstuff/docs/html/webby/a00019.html#gace404adfffdcfe00660631c1b74d141e">HTTP response has been generated</a> with an object representing the <a href="http://mosermichael.github.com/cstuff/docs/html/hutils/a00025.html">HTTP response header</a></li>
<li>A response data block or chunk <a href="http://mosermichael.github.com/cstuff/docs/html/webby/a00019.html#ga3fc7467c414925853cc55e085bef5ddc">has been generated</a> , one or more blocks of data that make up the HTTP response body, or a single object representing a chunk of the response, in case that chunked transfer encoding is used for the response.</li>
<li>An event that signals the <a href="http://mosermichael.github.com/cstuff/docs/html/webby/a00019.html#ga3424cf82a3e72ff1a900de9383c52e61">end of the HTTP response</a></li>
</ul>
<p>The more or less detailed documentation of this interface can be <a href="http://mosermichael.github.com/cstuff/docs/html/webby/a00019.html">found here</a>.</p>
<p>Now for each event, the filter object will process a different callback function. Many frameworks such as <a href="http://mina.apache.org/">Apache Mina</a> or <a href="http://www.jboss.org/netty">JBOSS Netty</a> will ask you to implement a generalized ‘protocol layer’ that passes around a generalized message object, this object has to fit all cases (The <a href="http://www.scala-lang.org/">Scala</a> language even has the construct of the <a href="http://www.scala-lang.org/node/107">case class</a> very big abstractions come out of it). I think that this is an over generalization. One of the virtues of doing C with classes is that it encourages the programmer to choose an abstraction that is not too far removed from the actual task, this is of course the result of the programmer being reluctant to type too much, because it is not too easy to declare a new class ;-)</p>
<h3 id="virtual-hosts">Virtual hosts.</h3>
<p>Each HTTP/1.1 request includes a Host header, the value of this protocol header is a domain name + optional port number; this allows the web server to implement ‘virtual hosting’;</p>
<p>WEBBY can work with two scenarios:</p>
<ul>
<li>No virtual hosts are defined, all filters form one processing chain; in this case the server checks for the presence of the Host header, but accepts any values.</li>
<li>There are virtual hosts defined. In this case the Host header has to mach one virtual host definition. Also each virtual host can have a different sequence of installed filters.</li>
</ul>
<h3 id="the-concept-of-the-servlets">The concept of the Servlets</h3>
<p>So where is the actual response generated?</p>
<p>A Http response can be generated by a filter, but many WEB servers offer a simplified interface for generating (dynamic) content, a Servlet object has one callback that receives the request header object, and all of the request data.</p>
<p>Java Servlets have the servlet; IIS has the ISAPI application ; and WEBBY will have its own HTTP_SERVLET interface.</p>
<p>The interface of the HTTP servlet is <a href="http://mosermichael.github.com/cstuff/docs/html/webby/a00020.html">documented here</a> , an example usage of WEBBY the web server with an example Servlet <a href="https://github.com/MoserMichael/cstuff/blob/master/webby/webbyimpl/test/embed_example.c">is here</a></p>
<h3 id="webby-as-a-standalone-application--dynamic-loading-of-filters-and-servlets">WEBBY as a standalone application / dynamic loading of filters and Servlets</h3>
<p>WEBBY can also exist as a standalone application; In this case it will load the filters and servlets dynamically from shared libraries. The shared library is expected to export a function that initializes the HTTP_FILTER object and returns it from the shared library. Filters are supposed to export the load_filter function and servlets export the load_servlet function. An example of a servlet as a shared library <a href="https://github.com/MoserMichael/cstuff/blob/master/webby/webbyimpl/test/helloworld.c">is here</a></p>
<p>The web server executable:
webycoro - the web server implementation with co-routines
webythread - the web server implementation with pthreads</p>
<p>Both programs have the same command line, which is <a href="https://github.com/MoserMichael/cstuff/blob/master/webby/webbyimpl/webbymain_blurb.txt">explained here</a></p>
<h3 id="what-comes-next">What comes next.</h3>
<p>Plan A) Obviously I should do load testing that compares the performance of pthread implementation versus performance of co-routine server with events. Unfortunately this requires access to some costly hardware, which I currently do not have; If somebody can kindly donate 5000-10000$ to me for this noble purpose? So for the meanwhile this task has to wait.</p>
<p>Plan B) Of course an area where the co-routine server would shine is web applications with database back end access. This would require some hacking - the database access libraries for various SQL database clients (namely Postgress/Mysql/Oracle) would have to be adapted to suite the EVCOROTHREAD package. Then, further this would have to be integrated with an embedded scripting language, <a href="http://www.lua.org/">Lua programming language</a> would fit the purpose. Anybody wants to server 10000 connections simultaneously, that do all require SQL queries/updates ?</p>
<p>So paradoxically such a system would have commercial potential. The only viable business model here would be dual Open source / commercial support option; The open source component would have the purpose of establishing trust / generating interest and is absolutely required. The problem of course if anybody would be prepared to shell out money for an HTTP server / application server in 2011 - given all the free solutions available; of course scalable back end access would be a strong selling point here.</p>
<p>Technically there is a risk here, does the <a href="http://www.lua.org/">Lua</a> runtime / LUA JIT allow for suspend / resume model or for low overhead JIT per request ? The <a href="http://www.erlang.org/">Erlang programming language</a> is supposed to have low overhead interpreter instances, but a functional programming language will probably not be too popular with developers, and also Erlang does not seem to have a JIT. Is Lua managing with small stack sizes, or does it need a huge stack?</p>
<p>What if non blocking event driven server does not have a sufficient practical advantage over connection per operating system thread model ? So at least some level of stress testing is required, before plan B is pursued. One thing I do strongly dislike of is to speculate on the subject of performance; so that’s exactly the reason why WEBBY has multiple implementations.</p>
<h3 id="still-further-deliberations-on-the-same-subject">Still further deliberations on the same subject</h3>
<p>I guess Lua would not be the right language ; for whatever reasons it is popular with embedded applications and games; also very special hacker applications like nmap and wiresharks use Lua as a scripting language. The rather conservative world of WEB development sticks with Java; Another reason is that Lua already has its own co-routines; and we have little to offer here.</p>
<p>There are of course, some lesser used but still wide spread tools like PHP, ColdFusion or ASP. One possibility is to implement such a niche system from scratch and to market a commercial solution for any of these tools; Such a product would receive only limited attention of a niche system, and in the long term it would probably be futile to invest in a fading platform; alas people would probably be more prepared to spend money on it.</p>
<p>Of course we could do a version for Java; I guess the most general solution here would be to create a customized JVM starting with the OpenJDK sources; the JVM would introduce the concept of co-routine threads;</p>
<p>So a regular Java thread will be able to host multiple co-routine, cooperatively scheduled threads;
A regular thread object will then have a field that says if a co-routine is currently executed.</p>
<p>Now when the application may well call one of the java monitor synchronization primitives of class java.lang.Object wait/notify/notifyall; if a co-routine thread is currently executed, then a special version of these function has called, one that does not block the co-routine thread completely, wait would yield control, notify would resume the co-routine; of course both wait and notify have to be called from a co-routine that is invoked as part of the current operating system thread.</p>
<p>Also specialized versions of java.net.Socket and their nio counterparts will be implemented, those will suspend the current co-routine thread when the socket operation blocks; just like the <a href="http://mosermichael.github.com/cstuff/all/projects/2011/10/09/Non-Blocking-IO-with-coroutines.html">EVCOROTHREAD</a> package does.</p>
<p>This is a fine design, but I am not quite sure if I want to donate my effort to <a href="http://en.wikipedia.org/wiki/Larry_Ellison">Mr. Ellison</a> for free, without charge, after all the guy already has a <a href="http://en.wikipedia.org/wiki/Rising_Sun_(yacht)">very big yacht</a> in the world.</p>
Non blocking IO with co-routines
2011-10-09T00:00:00+00:00
http://mosermichael.github.com//all/projects/2011/10/09/Non-Blocking-IO-with-coroutines
["", "projects"] | <h2 id="non-blocking-io-with-co-routines">Non blocking IO with co-routines</h2>
<p class="publish_date">
9 October 2011
</p>
<p>This project extends the co-routine library and so that it will be useful with non blocking networking.</p>
<p>There is the long standing debate on how to write a server, the basic choices are</p>
<ul>
<li>The server maintains an operating system thread per network connection connection.</li>
<li>One operating system thread can handle multiple network connections. The thread watches a set of non blocking connections for ‘read events’; Every now and then a particular connection will receive new data over the network, this constitutes a ‘read event’; The connection has to handle the read event by consuming all available data, this way the the read event shall be cleared; when a response is ready, it is sent over the network connection, it may happen that the response is larger than what can be transmitted immediately; in this event the send system call fails, but indicates that it can be resumed later on; the connection must now register interest in receiving the ‘write event’ - once received it will signal that more data can be sent over the network connection. Once all data has been sent, interest in the ‘write event’ must be cleared.</li>
</ul>
<p>Now implementing non blocking servers is tricky; usually there is a ‘event loop’ or ‘Reactor’ component that handles the details of watching network connections for events; now when a new event must be handled a callback functions is called by the ‘event loop’/’Reactor’.</p>
<p>As stated earlier in the co-routine article, chopping up the server logic into a state machine that fits the callback functions can turn into many bugs, sweat and tears. So this project integrates the <a href="http://mosermichael.github.com/cstuff/all/projects/2011/10/05/Co-Routines.html">co-routine package</a> with the <a href="http://libevent.org/">libevent</a> event loop.</p>
<p>Here are the abstractions that make up the EVCOROTHREAD library</p>
<ul>
<li>EVLOOP - the event loop / reactor, once this object is initialized, the EVLOOP_run method is called, from this point on the event loop will continuously listen for io events and service io events.</li>
</ul>
<!-- -->
<ul>
<li>EVTHREAD - the co-routine thread that handles one or more network connections. The application can also explicitly create a EVTHREAD, a co-routine thread owns one or more EVSOCKET objects, when the co-routine thread exits, then all EVSOCKET objects owned by it are closed. The subroutine thread can call services of EVSOCKET object, while it is running.</li>
</ul>
<!-- -->
<ul>
<li>EVSOCKET - the network connection, each network connection belongs to an EVTHREAD co-routine, this objects has methods to connect / read / write a EVSOCKET object; when each one of these methods blocks - that means it cannot be completed immediately, then the current co-routine thread is suspended, and control returns to the event loop thread; the event loop will then pick another co-routine thread where one of its sockets have a pending event, and run it.</li>
</ul>
<!-- -->
<ul>
<li>EVTCPACCEPTOR - the object that accepts new incoming connection and wraps it into a EVSOCKET object; a co-routine thread is created per connection and is started immediately.</li>
</ul>
<h3 id="the-abstractions-in-more-detail">The abstractions in more detail</h3>
<p>In this section each class will be covered in more detail, and a TCP echo server is also presented in the process.</p>
<h4 id="initializing-the-event-loop">Initializing the event loop</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>01: #include <corothread/evthread.h>
02:
03: STACKS stacks;
04: EVLOOP *loop;
05:
06: CTHREAD_libinit()
07:
08: STACKS_init( &stacks, 100 , 5 ) );
09:
10: loop = EVLOOP_init( &stacks );
</code></pre></div></div>
<p>Line 6 initializes the co-routine library; line 8 initializes the allocator of co-routine stacks; here a limit of 100 co-routines each with 4 usable memory pages each can be created; there cannot be more than 100 co-routine threads serviced in this example, this is a convenient way to impose a limit / throttle the number of connections that can be created ; very important feature.
Line 10 creates the event loop object - the stack allocator is passed to it.</p>
<h4 id="creating-socket-listener-for-incoming-connections">Creating socket listener for incoming connections</h4>
<p>The EVTACCEPTOR class encapsulates a passive socket that is listening for incoming connections.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>01: EVTCPACCEPTOR *acceptor;
02: IPADDRESS addr;
03: SOCKADDR saddr;
04:
05: IPADDRESS_any( &addr, AF_INET );
06: SOCKADDR_init( &saddr, &addr, 9999 );
07:
08: acceptor = EVTCPACCEPTOR_init_ex( loop, &saddr, 30, echo_thread_factory, 8192, 8192 );
</code></pre></div></div>
<p>Lines 5 and 6 initialize the socket address that will listen for incoming connections at any interface on port 9999.</p>
<p>Line 8 creates the listener object - EVTCPACCEPTOR .
the first parameter loop is a reference to the event loop
parameter saddr is the address that will be listened at
30 - the sizeof the backlog parameter passed to listen system call.
echo_thread_factory - once a new connection has been accepted, this callback function is called ; it will determine the thread procedure of the newly created thread, and will also create a context pointer that holds the data of the new thread procedure.
8192, 8192 - respectively the size of the socket receive and send buffers; a value of -1 says that the default size is not changed.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>01: int echo_thread_factory (int fd, EVTHREAD_PROC *proc, void **ctx )
02: {
03: *proc = echo_thread;
04: *ctx = 0;
05: return 0;
06: }
07:
08: void echo_thread( EVTHREAD *thread, struct tagEVSOCKET *socket, void *user_ctx)
</code></pre></div></div>
<p>the echo_thread_factory sets the thread procedure value to echo_thread. The context value is set to 0, meaning that user_ctx parameter of echo_thread will be zero.
If echo_thread_factory returns non zero value, then the accepted socket is closed immediately.</p>
<h4 id="the-co-routine-thread">The co-routine thread</h4>
<p>Once a new connection has been accepted, a co-routine is started and the echo_thread procedure is invoked.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>01: static struct timeval TIME_OUT = { 3 , 0 };
02:
03: void echo_thread( EVTHREAD *thread, struct tagEVSOCKET *socket, void *user_ctx)
04: {
05: char *buffer;
06: int rt;
07:
08: buffer = malloc( BF_SIZE);
09:
10: while ( (rt = EVSOCKET_recv( socket, buffer, BF_SIZE, 0, TIME_OUT)) > 0) {
11: if ( EVSOCKET_send( socket, buffer, rt, 0, TIME_OUT ) != rt ) {
12: break;
13: }
14: }
15:
16: free(buffer);
17: }
</code></pre></div></div>
<p>The echo_thread procedure receives the following arguments</p>
<ul>
<li>thread - pointer to EVTHREAD the current co-routine thread.</li>
<li>socket - if a socket is initially attached to the thread, as is the case when receiving a new connection via EVTCPACCEPTOR, then this value points to the connection object that has been created by EVTCPACCEPTOR</li>
<li>user_ctx - pointer to user context as returned by echo_thread_factory procedure.</li>
</ul>
<p>At line 10 we are actually reading some data from the connection. EVSOCKET_recv is the function to do it; it also receivess a TIME_OUT (struct timeval) parameter that denotes the read timeout.</p>
<p>EVSOCKET_send sends the return back via the connection, note that it too receives a timeout parameter.</p>
<p>Note that the buffer used here is allocated from the heap; great care should be taken with stack allocations, especially with 32 bit code, the stack is usually much smaller than what you would expect of pthread threads; (see the co-routine article for more details).</p>
<h4 id="delays-and-timing">delays and timing</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>01: void evthread_proc( EVTHREAD *thread, EVSOCKET *socket, void *user_data )
02: {
03: struct timeval tv;
04:
05: tv.tv_sec = 3;
06: tv.tv_usec = 0;
07:
08: EVTHREAD_delay( thread, tv );
09: }
</code></pre></div></div>
<p>EVTHREAD_delay will pause the current co-routine for three seconds; during this time other co-routine threads will be active.</p>
<h3 id="download-instructions">Download instructions</h3>
<p>Get <a href="https://github.com/MoserMichael/cstuff">My stuff</a>
cd build
./build-evcorothread.sh
(For platform specific stuf ./build-coroutine.sh ARCH=x86_64)
For tutorial see <root direcotory="">/corothread/tutorial</root></p>
<p>Set inlclude search path to <root directory=""></root></p>
<p>Add the following libraries
-lcutils -lbutils -lnutils -lcorothread -levcorothread -pthread</p>
<p>If one wants to use shared libraries add
-lcutilssh -lbutilssh -lnutilssh -lcorothreadsh -levcorothreadsh -pthread</p>
<h3 id="limitations-of-this-approach">limitations of this approach</h3>
<p>Co-routines can simplify development of non blocking servers; however there are situations when this technique can’t be used. Note that for a connection we will either receive data (SOCKET_recv) or we will be writing data to it (SOCKET_send) ; we can’t do both at the same time. So you will not be able to do applications that stream data in both directions - from server to client and from client to server.</p>
<h3 id="what-comes-next">what comes next</h3>
<p>In the next part of this series we will build a small HTTP server, and do so with three paradigms</p>
<ul>
<li>Operating system thread per connection</li>
<li>Co-routine thread per connections</li>
<li>regular libevent - multiple connections per thread with event callbacks / Reactor pattern.</li>
</ul>
<p>The HTTP server will be embeddable - an application program will be able to run the WEB server as a library. It will then be interesting to study performance of these three models.
Apart from that it will be a good tool in order to test and debug the library.</p>
Simple co-routine package in C
2011-10-05T00:00:00+00:00
http://mosermichael.github.com//all/projects/2011/10/05/Co-Routines
["", "projects"] | <h2 id="simple-co-routine-package-in-c">Simple co-routine package in C</h2>
<p class="publish_date">
5 October 2011
</p>
<p>This project presents a simple user mode threading library, it turns out that this concept has many names</p>
<ul>
<li><a href="http://en.wikipedia.org/wiki/Coroutine">co-routine package</a></li>
<li><a href="http://en.wikipedia.org/wiki/Continuation">continuations</a> .</li>
<li>In the Windows world this concept is also called a <a href="http://en.wikipedia.org/wiki/Fiber_(computer_science)">Fiber</a></li>
<li>Also known as user mode threading</li>
<li><a href="http://en.wikipedia.org/wiki/Computer_multitasking#Cooperative_multitasking.2Ftime-sharing">cooperative multitasking</a></li>
</ul>
<p>It is a small wonder how so many computer science terms all denote the the very same concept. Usually a confused language is either a sign that the same concept has been reinvented several times, or that something is not very well understood, this fact is known since the times of Babylon. Lets see.</p>
<p>Software engineers try to create complex systems out of simple parts; each simple part implements some sequence of commands - a process of sorts. Let’s look at a simple example; process (A) creates a sequence of Fibonacci numbers, and process (B) receives each Fibonacci number and computes the sum of the square roots of each Fibonacci numbers.</p>
<p>Now one has two choices, either process (A) and process (B) share the same stack, or they do not share the same stack.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>void make_fibonacci_sequence( uint64_t *fib, size_t num_fib )
{
uint64_t a=0, b=1, n;
size_t i;
for(i=0; i < num_fib; i++, a = b, b = n) {
n = a + b;
fib[ i ] = n;
}
}
double sum_of_roots( uint64_t *num, size_t num_size )
{
size_t i;
double sum_roots = 0;
for( i = 0; i < num_size; i++)
sum_roots += sqrt( (double) num[i] );
return sum_roots;
}
int main()
{
uint64_t num[MAX_NUM];
make_fibonacci_sequence( num, MAX_NUM );
printf("Sum of roots of fibonaci numbers: %f\n", sum_of_roots( num, MAX_NUM ) );
}
</code></pre></div></div>
<p>In this example both steps share the same stack, first Fibonacci numbers are computed and stored in an array by the first procedure, and then the result is passed to the second procedure that computes the sum of the square roots.</p>
<p>Now what if the array of results is very large and we are short on computer memory ? What if one has to know the current sum of square roots immediately upon receiving a new Fibonacci number / react to the event of a new Fibonacci number?
In this situation both processes have to share the same stack, so that each new Fibonacci number is passed to a callback function <a href="http://en.wikipedia.org/wiki/Callback_(computer_programming)">callback function</a></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>typedef void (*FIBONACCI_CB) (uint64_t num, void *ctx);
void generate_fibonacci_sequence( size_t num_size, FIBONACCI_CB callback, void *ctx )
{
uint64_t a=0, b=1, n;
size_t i;
for(i=0; i < num_size; i++, a = b, b = n) {
n = a + b;
callback( n, ctx );
}
}
void fibonacci_cb( uint64_t fib_num, void *ctx)
{
double *fibb_sum = (double *) ctx;
*fibb_sum += sqrt( (double) fib_num );
}
int main()
{
uint64_t num[MAX_NUM];
double result = 0;
generate_fibonacci_sequence( MAX_NUM, fibonacci_cb, &result );
printf("Sum of roots of fibonaci numbers: %f\n", result );
}
</code></pre></div></div>
<p>.
This mode of doing things has a marked disadvantage: The procedure of computing the sum of roots is now chopped up into multiple calls of the callback function fibonacci_cb ; on each call this function has to retrieve the current state of the computation from the argument ‘context parameter’ , perform a processing step and then update the result back into the ‘context parameter’. A more complex process also must be partitioned into a state machine, the state is maintained extra in the ‘context parameter’. No fun.</p>
<p>So one need sto have both processes running one after the other; first a number is produced , and then one wants to switch to consuming it; alas each process should be able to run in parallel - as if it were a sequence of steps that does not bother of the other process at all.</p>
<p>So each procedure needs its own stack. Now the same basic problem is being solved in a multitude of ways; each solution solves the problem for some specific context,</p>
<ul>
<li>multi processing: Two processes are connected via a pipe; the Fibonacci number generator writes its result to the pipe; the square root adder reads each number off the pipe.</li>
<li>Multi-threading with shared state;</li>
<li>actor model - several threads communicate via known queues; there is no shared state between the threads other than the queues; all communication between the threads is strictly via the queues; each message is not modifiable.</li>
<li>Python / Perl generator functions (co -routines implemented by the interpreter run-time).</li>
</ul>
<p>Now co-routine will simply maintain multiple stacks per procedure; the Fibonacci number generator function has its own stack, the function that consumes Fibonacci numbers has its own stack. When a new number is computed, Fibonacci number generator switches the stack to the consumer of numbers, when the consumer is ready and wants to get a new number then it switches the stack to the generator function.</p>
<p>There is a significant difference - with multiple threads or multiple processes it is the operating system that decides which task will run next; with co-routine the application decides when to switch between stacks.</p>
<p><a name="pf"></a></p>
<h3 id="performance-advantages">Performance advantages</h3>
<p>Another motivation for co-routines is performance. Lets say we have the following example that creates 10000 operating system threads with the pthread package, where all threads are put to sleep.</p>
<p>With Linux (2.6.38-8) you will see that responsiveness of the system will decrease dramatically; The top utility will display a CPU usage of 30% and more and the load average (a somewhat obscure statistic that sums up the system utilization) will increase dramatically, all within a very short time.</p>
<p>How could that be explained? The Linux scheduler maintains several run queues - each run queue is a data structure (not necessarily a queue in that sense) that maintains each tasks (processes or threads) of a given priority class; overall there are 255 priority classes. Now on Linux a run queue holds all tasks; those that are runnable and those that are sleeping; this implies that at the time when the operating system scheduler is deciding which task to run next, it has to consider all tasks of a given priority class. That implies that the Linux kernel is probably not too good at scheduling a large number of threads;</p>
<p>Another issue is that with cooperative multitasking a switch of context occurs when the application decides that such as switch must be done; so a context switch is performed when one is needed; On the other hand, a preemptive scheduler has to maintain a lot of other consideration like fairness of tasks, system throughput, responsiveness of the system, etc. etc. so the decision to switch contexts will be taken at a different time; not at exactly the same spot when one is required by the task at hand.</p>
<p>Please note that an operating system that maintains separate queues for sleeping tasks as opposed to runnable tasks will probably be much better at the task of scheduling a huge number of threads; BSD appears to do so, alas I currently don’t have access to the required hardware / software in order to do sufficient benchmarks; etc. etc.</p>
<p>Historically high performance systems where all done with user mode scheduled threads; CICS does so - which is of course a very important system on the Mainframe; and this system is of course still very much in use for certain tasks.</p>
<p>Some more obscure details on running the test that creates 10000 threads via pthreads.
make sure that the system wide limit on the number of tasks does not bother this test; you can display the current limit by running ‘‘cat /proc/sys/kernel/threads-max”; the root/superuser can change this value at run time, one does not have to restart the system for that. An enterprise system
Make sure that the current soft limit does not limit the number of tasks that can be created; the command “ulimit -u” will display the current value of a shell;
the test application creates a short stack for each thread, this way a 32 bit process will not run out of virtual memory address space (by default a pthread stack has 10 MB).</p>
<h3 id="the-cthread-package">The CTHREAD package</h3>
<p>The remainder of this article will introduce a cooperative user-mode threading package; which is implemented as a library of functions callable by any C program on Linux, alas portable to any POSIX compliant operating system;
This package will be named the CTHREAD package.</p>
<h3 id="how-to-pass-a-typed-set-of-values-around">How to pass a typed set of values around.</h3>
<p>This library will be passing parameters from and to the co-routine thread, so here is a utility class in C for packaging of a typed set of values.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#include <corothread/val.h>
VALUES values;
uint8_t n8 = 250, _n8;
uint16_t n16 = 65000, _n16;
int32_t n32 = 42, _n32;
int64_t n64 = 0xBEEFBEEFBEEFBEEF, _n64;
char *sval = "aaa", *_sval;
VALUES_init( &values )
VALUES_print( &val, "%hhu%hd%d%qd%s", n8, n16, n32, n64, sval );
VALUES_scan( &val, "%hhu%hd%d%qd%s", &_n8, &_n16, &_n32, &_n64, &_sval );
</code></pre></div></div>
<h3 id="more-on-stacks">More on stacks</h3>
<p>A co-routine stack consists of a continuous number of memory pages; now at the top of the stack there is a memory page that is not accessible, any attempt to read or write to this memory page will result in a SIGSEGV - access violation signal. This turns out to be a very nice trick, when the co-routine stack grows out of proportion / overflows, then one will immediately get to know that something got wrong.</p>
<p>There are times when an application will want a very large number of co-routines - in the order of tens of thousands; Now on 32 bit machine architectures one will choose a very small stack, otherwise one will run out of virtual memory to address;</p>
<p>Please note that with small co-routine stacks one should not allocate big arrays on the co-routine stack.</p>
<p>The CTHREADS package implements a pool of stacks, so that creation of co-routine will be very fast and will usually not even require even a single system call. With this purpose there is a STACKS reserves one very big continuous are of virtual memory addresses and carves the application stacks out of this memory are. The STACKS object has to be passed to co-routine thread creation function.</p>
<h3 id="fibonacci-example-with-co-routines">Fibonacci example with co-routines</h3>
<p>The same example that computes a sequence of Fibonacci numbers now by using the CTHREAD co routine package. The co-routine make_fibonacci_sequence now generates a number and calls CTHREAD_yield to switch the stack back to the caller of the co-routine, and to pass the computed value to the caller.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#include <corothread/cthread.h>
#include <stdio.h>
#include <assert.h>
#include <math.h>
#define NUM_STACKS 10000
#define MAX_NUM 90
// -----------------------------------------------------------------
void make_fibonacci_sequence(VALUES *values)
{
int i, max_num;
void *val;
uint64_t a,b,n;
// retrieve argument passed to co-routine - how many iterations to run.
VALUES_scan( values, "%d", &max_num );
for(a = 0, b = 1, i = 0; i < max_num; i++) {
n = a+b;
// now that a new Fibonacci value has been computed, it can be passed to the caller;
// the next function switches stacks from co-routine back to the caller.
CTHREAD_yield( 0, "%qu", n );
a=b;
b=n;
}
}
double sum_of_roots( size_t max_num, STACKS *stacks )
{
CTHREAD *th;
VALUES *values;
uint64_t fib;
int i;
double sum_roots = 0;
// initialize the co-routine thread
th = CTHREAD_init( stacks, make_fibonacci_sequence );
assert( th != 0 );
// start coroutine. pass the value of max_num to the coroutine.
assert( CTHREAD_start( th, &values, "%d", max_num ) == 0 );
// on return from CTHREAD_start, the return value from the coroutine is now stored in the values object.
for( i = 0 ; i < max_num && CTHREAD_state(th) != CTHREAD_STATE_EXIT; i++) {
VALUES_scan( values, "%qu", &fib );
sum_roots += sqrt( (double) fib );
// resume the co-routine thread, switch stack to co-routine thread.
CTHREAD_resume( th, &values, 0 );
// on return from CTHREAD_resume, the return value from the co-routine is now stored in the values object.
}
// free co-routine object.
assert( ! CTHREAD_free( th ) );
return sum_roots;
}
int main()
{
STACKS stacks;
// library first time initialisation
assert( ! CTHREAD_libinit() );
// initialize the pool of stacks - here one stack that takes five memory pages is allocated.
assert( ! STACKS_init( &stacks, 1 , 5 ) );
printf("Sum of roots of Fibonacci numbers: %f\n", sum_of_roots( MAX_NUM, &stacks ) );
// release pool of stacks
assert( ! STACKS_destroy( &stacks ) );
return 0;
}
</code></pre></div></div>
<h3 id="download-instructions">Download instructions</h3>
<p>Get <a href="https://github.com/MoserMichael/cstuff">My stuff</a>
cd build
./build-coroutine.sh
(For platform specific stuf ./build-coroutine.sh ARCH=x86_64)
For tutorial see <root direcotory="">/corothread/tutorial
Add the following libraries -lcorothread -lcutils -lnutils -lpthread
If you want to use shared libraries then use -lcorothreadsh -lcutilssh -lnutilssh -lpthread</root></p>
<h3 id="what-comes-next">What comes next</h3>
<p>The next article in this series will show how to handle IO events / non blocking sockets with co-routines; For this purpose a library will be presented that invokes a co-routine thread per new incoming connection; the co-routine will use blocking primitives in order to receive data from and write to the connection; when a connection blocks ; the library will switch the stack and handle an IO event with another co-routine.</p>
Fixing tricky C/C++ compilation errors
2011-09-16T00:00:00+00:00
http://mosermichael.github.com//all/projects/2011/09/16/preprocessor
["", "projects"] | <h2 id="fixing-tricky-cc-compilation-errors">Fixing tricky C/C++ compilation errors</h2>
<p class="publish_date">
17 July 2011
</p>
<p>Both C and C** share the same preprocessor; a preprocessor transforms the source code before it is compiled; Now many compilation errors are due to the programmer having misunderstood how the preprocessor changed the text of the source file.</p>
<p>Luckily with GCC and other compilers (aCC for instance), it is possible to see the preprocessor output as is, before it is passed on to the compiler.</p>
<ul>
<li>On both gcc and aCC (the HP C compiler) you can just add the -E option to the compiler ; now the compiler will not compile the program, only preprocessor does something, and the output of the preprocessor step is written to standard output.</li>
<li>With gcc you can use compiler option -save-temps; with this option all intermediate results (preprocessor result, assembly listing) will be stored into separate files; For example</li>
</ul>
<!-- -->
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcc -save-temps example.cpp -o example.o
</code></pre></div></div>
<p>After running this you will have example.ii - with output of the preprocessor right in the same directory as example.cpp (also example.s has the assembly listing).
The output of the preprocessor is <a href="http://gcc.gnu.org/onlinedocs/cpp/Preprocessor-Output.html">described here</a> Now it is difficult for a human to parse this text;</p>
<p>So it’s time for a tool that would annotate the preprocessor output.</p>
<ul>
<li>So for each line of the preprocessor output I would like to add the source file + line number; so given a compilation error I can easily find the line number that has caused it.</li>
<li>For each line of the preprocessor output I would like to have the text of the source file that caused it, If these two data points are different, I would like to see them both at the same spot.</li>
</ul>
<p>That’s exactly what we are doing with a nice Perl script, welcome to the <a href="https://github.com/MoserMichael/cstuff/blob/master/scripts/cppbeautifier">cppbeautifier script</a></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Usage:
cppbeautifier -i <cpp_file> [-d <directory>] -o <report_file>
Description
Takes an output file produced by the c processor and anotatets it,
so that it will be readable by humans.
For each line of preprocessor output we find the corresponding file from
the source file; if these are different then both source and preprocessor
line are displayed, this way it it is easier for the programmer to determine
the result of the macro expansion.
Lines from the source file have prefix <filename>:<line number>:+ and lines
from preprocessor output have prefix <filename>:<line number>:-
The preprocessor output format is described here:
http://gcc.gnu.org/onlinedocs/cpp/Preprocessor-Output.html
Please note that the preprocessor often includes reference to source
files with their relative path, so you should either use this program from
the same current directory where you have run the preprocessor / compiler.
Alternatively set the -d option to this directory.
</code></pre></div></div>
<p>Lets show off with some example output: here a generator macro is creating a function, and the script output shows us how the preprocessor output correlates to the source macro invocation. Lines with + are from the source file, lines with - are the expanded macro - preprocessor output.</p>
<pre>
# 167 "bf.h"
bf.h:167:+:BF_DECLARE_PUT_MACRO( uint8_t )
bf.h:167:-:static inline int BF_put_uint8_t ( BF *bf, uint8_t val ) { if ((bf->end - bf->put_pos) < (ptrdiff_t) sizeof( uint8_t ) ) { return -1 ; } do { uint8_t nval = (val); *( ( uint8_t *) bf->put_pos) = nval; bf->put_pos += sizeof ( uint8_t ); } while(0); return 0; }
bf.h:168:+:BF_DECLARE_PUT_MACRO( int8_t )
bf.h:168:-:static inline int BF_put_int8_t ( BF *bf, int8_t val ) { if ((bf->end - bf->put_pos) < (ptrdiff_t) sizeof( int8_t ) ) { return -1 ; } do { int8_t nval = (val); *( ( int8_t *) bf->put_pos) = nval; bf->put_pos += sizeof ( int8_t ); } while(0); return 0; }
bf.h:169:+:BF_DECLARE_PUT_MACRO( uint16_t )
bf.h:169:-:static inline int BF_put_uint16_t ( BF *bf, uint16_t val ) { if ((bf->end - bf->put_pos) < (ptrdiff_t) sizeof( uint16_t ) ) { return -1 ; } do { uint16_t nval = htons( val ); *( ( uint16_t *) bf->put_pos) = nval; bf->put_pos += sizeof ( uint16_t ); } while(0); return 0; }
bf.h:170:+:BF_DECLARE_PUT_MACRO( int16_t )
bf.h:170:-:static inline int BF_put_int16_t ( BF *bf, int16_t val ) { if ((bf->end - bf->put_pos) < (ptrdiff_t) sizeof( int16_t ) ) { return -1 ; } do { int16_t nval = htons( val ); *( ( int16_t *) bf->put_pos) = nval; bf->put_pos += sizeof ( int16_t ); } while(0); return 0; }
bf.h:171:+:BF_DECLARE_PUT_MACRO( uint32_t )
bf.h:171:-:static inline int BF_put_uint32_t ( BF *bf, uint32_t val ) { if ((bf->end - bf->put_pos) < (ptrdiff_t) sizeof( uint32_t ) ) { return -1 ; } do { uint32_t nval = htonl( val ); *( ( uint32_t *) bf->put_pos) = nval; bf->put_pos += sizeof ( uint32_t ); } while(0); return 0; }
bf.h:172:+:BF_DECLARE_PUT_MACRO( int32_t )
bf.h:172:-:static inline int BF_put_int32_t ( BF *bf, int32_t val ) { if ((bf->end - bf->put_pos) < (ptrdiff_t) sizeof( int32_t ) ) { return -1 ; } do { int32_t nval = htonl( val ); *( ( int32_t *) bf->put_pos) = nval; bf->put_pos += sizeof ( int32_t ); } while(0); return 0; }
</pre>
<h3 id="a-short-and-probably-incomplete-history-of-macro-facilities">A short (and probably incomplete) history of macro facilities</h3>
<p>Macro facilities have a common problem - they just transform text / rewrite text from one form into another; so the macro facility does not know anything about the syntax of the text that it is being transformed. For simple usage - conditional compilation of certain blocks of text, this ok; for macros that rewrite the source into something else we will have to look separately at the macro text, and at the result of the macro expansion.</p>
<p>So Macro facilities were the preferred solution with languages that have no or little syntax;</p>
<ul>
<li>The assembly language of S/390 ; Back in the days of the mainframe, people were writing very complicated application programs all in assembly. Now the cost of leasing processor cycles was very high; so organizations preferred to keep a lot of assembly programmers rather than compromising for an application that would be slower at runtime. Now the IBM assembler has a very strong macro facility; in fact all system calls were macro invocations (there is no such thing as a runtime library on the mainframe); Applications were regularly doing huge macros that were expanding into hundreds and thousands of lines of code; Now macros where a central feature of the language; in order to debug them the assembler would produce a fairly huge assembly listing; all levels of macro invocations where listed along with how they expanded the text / one could exactly see what text was being modified by which macro invocation. It is interesting that this feature of exactly listing the macro invocation input versus its output seems to have been lost thereafter.</li>
</ul>
<p>* LISP (or Lisp) macros; Lisp doesn’t have a syntax; in fact a Lisp program is sort of a textual representation of a Abstract Syntax tree, in this minimal language the macro facility was very strong, a macro invocation can do computations of its own; Macros are the preferred way of extending the language; for example Common Lisp does not have a loop construct, instead there is a built in loop macro. Lisp Macros are the preferred way of doing ‘internal DSL’ - that means an extension of the syntax of language A by means of features already present in the language A. A lot of wizardry and hacking has been done with lisp macros. More on the <a href="http://www.dreamsongs.com/NewFiles/HOPL2-Uncut.pdf">history of Lisp can be read here</a> . With Lisp you have the macroexpand and macroexpand-1 functions; macroexpand will show you the full result of expanding a macro, macroexpand-1 will show you the expansion of a macro to the next level (macro’s within macros are not expanded with macroexpand-1).</p>
<ul>
<li>The C preprocessor; The C preprocessor is less powerful by comparison than what has been, in fact C is a very minimal language by itself. One strong but often overlooked feature of the C preprocessor is that it facilitates the constructing of code generators that embed C statements; example of such systems are YACC/BISON compiler generator or various embedded SQL solutions for instance (Oracle PSQL for example); The higher order language can include sections written in C that do something; now this text is translated into C; but the debugger knows exactly how the embedded C statement maps back to the original higher order language definition. This is done by means of <a href="http://gcc.gnu.org/onlinedocs/cpp/Line-Control.html">the #line directive</a> - the #line directive tells the compiler that the following section of code should be marked in the debugger line table as coming from a different source; the debugger is looking at the line table and will look for the source of this goodness in the higher order language definition.
Now with GCC and other compilers you can invoke the preprocessor as a separate process; this is done because most compiler stages are done as a processes that talk to next compiler stage via a pipes, so the preprocessor is just a command line program that pipes its output to the syntax checker / AST builder.</li>
</ul>
<!-- -->
<ul>
<li>Modern languages; Java, Scala most JVM languages - do not have a preprocessor, and they are proud of that (probably that is a feature that has been proudly inherited from Smalltalk). One of the better features of Java is it’s very fast compilation speed and lack of linkage step, this is achieved by integrating the semantic verification step of the compiler with the class loading mechanism; when the compiler checks for a method signature of a given class, it just looks up the meta data of a class in the already compiled class file. I guess a preprocessor would slow compilation speed, so Java people would not like that feature. Of course you lose some better features - like the C preprocessor support for code generators, well I guess one can’t always win; the C preprocessor is better integrating ‘external DSL’s’ - so be it. In Scala one can still do an ‘internal DSL’ - that means an extension of the language syntax of language A by means of features already present in the language A has; Now interestingly modern language extension mechanisms are not founded on any macro facilities; instead on this, we are now supposed to overload operators or work with monad comprehension (some syntactic trickery of combining functions like Lisp functions named map, filter and fold - that’s my explanation of monads : -)</li>
</ul>
jstack - the missing manual
2011-08-29T00:00:00+00:00
http://mosermichael.github.com//all/projects/2011/08/29/jstack
["", "projects"] | <h2 id="jstack---the-missing-manual">jstack - the missing manual</h2>
<p class="publish_date">
29 August 2011
</p>
<p>The <a href="http://linux.die.net/man/1/jstack-java-1.6.0-openjdk">jstack</a> utility is part of <a href="http://en.wikipedia.org/wiki/HotSpot">Hotspot JVM</a>, it is a command line program that obtains the current stack trace of all java threads. Actually it is one of the strong points of java that you can always get a stack trace for all threads of a process and also display all currently held locks. With ‘unmanaged’ c programs it is not quite so easy, not at all.</p>
<p>So when you are running jstack on Linux (or any other Unix) then there are some details that are missing from the documentations</p>
<ul>
<li>The jstack utility and JVM have to be from the same version of the JDK. If you have several versions of java installed then go figure.</li>
<li>The jstack utility and traced java process should run under the same user account. This needs some explanation</li>
</ul>
<p>Let us trace the system calls for running jstack on process id 10015</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>strace -f -oo /usr/java/jdk1.6.0/bin/jstack -l 10015
grep -F 'connect(' o
11558 connect(6, {sa_family=AF_FILE, path="/tmp/.java_pid10015"}, 110) = 0
11558 connect(6, {sa_family=AF_FILE, path="/tmp/.java_pid10015"}, 110) = 0
</code></pre></div></div>
<p>Interestingly jstack uses a <a href="http://en.wikipedia.org/wiki/Named_pipe">named pipe</a> with the file name of /tmp/.java_pid<processed> in order to communicate with the JVM of the traced process. Communication via the named pipe goes through a simple string based protocol</processed></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>11619 connect(6, {sa_family=AF_FILE, path="/tmp/.java_pid10015"}, 110) = 0
11619 write(6, "1", 1) = 1
11619 write(6, "\0", 1) = 1
11619 write(6, "threaddump", 10) = 10
11619 write(6, "\0", 1) = 1
11619 write(6, "-l", 2) = 2
11619 write(6, "\0", 1) = 1
11619 write(6, "\0", 1) = 1
11619 write(6, "\0", 1) = 1
...
11619 read(6, "0", 1) = 1
11619 read(6, "\n", 1) = 1
11619 read(6, "2011-08-28 11:17:49\nFull thread "..., 128) = 128
11619 write(1, "2011-08-28 11:17:49\nFull thread "..., 128) = 128
11619 read(6, "95e5400 nid=0x2aef waiting on co"..., 128) = 128
11619 write(1, "95e5400 nid=0x2aef waiting on co"..., 128) = 128
11619 read(6, "ne\n\n\"DestroyJavaVM\" prio=10 tid="..., 128) = 128
11619 write(1, "ne\n\n\"DestroyJavaVM\" prio=10 tid="..., 128) = 128
11619 read(6, "Locked ownable synchronizers:\n\t-"..., 128) = 128
11619 write(1, "Locked ownable synchronizers:\n\t-"..., 128) = 128
11619 read(6, "va.lang.Thread.State: TIMED_WAIT"..., 128) = 128
11619 write(1, "va.lang.Thread.State: TIMED_WAIT"..., 128) = 128
11619 read(6, "va.util.concurrent.locks.Abstrac"..., 128) = 128
</code></pre></div></div>
<p>Aha, so jstack just sends a string command to the traced JVM, and the JVM returns all report lines over the same named pipe’s socket.</p>
<p>So while the program is running, we can look at the named pipe</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>stat /tmp/.java_pid10015
Size: 0 Blocks: 0 IO Block: 4096 socket
Device: fd00h/64768d Inode: 14699683 Links: 1
Access: (0600/srw-------) Uid: ( 315/ mmoser) Gid: ( 231/ sps)
Access: 2011-08-28 11:17:07.000000000 +0300
Modify: 2011-08-28 11:13:20.000000000 +0300
Change: 2011-08-28 11:13:20.000000000 +0300
</code></pre></div></div>
<p>The access permissions say that only one account, the user account that created the java process, can access the named pipe. Even the root user can’t talk to this named pipe directly.</p>
<h3 id="a-script-to-automate-usage-of-jstack">A script to automate usage of jstack</h3>
<p>All this can be obtained from system information, just by knowing the process id number.</p>
<ul>
<li>Can see who is running the process, the permissions and owner of the /proc/[pid]/ directory say so (the linux operating system maintains this directory per running process )</li>
<li>The /proc/[pid]/maps includes all shared memory ranges used by the process, so each shared library has it’s text/code segment here + full path of the executable that is mapped in.
Each Hotspot java process loads a library called libjvm.so - so by finding the path name of libjvm.so we can infer the path of the used JDK version.</li>
</ul>
<p>The script that can do all this <a href="https://github.com/MoserMichael/cstuff/blob/master/scripts/jstack/do-jstack">is here</a>
It should be used as follows:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Usage: ./do-jstack <pid> [ -m -F ]
Description
The script performs the following steps
- check if <pid> is that of a running process
- check if <pid> is running from the same user account as current account
- check if <pid> is that of a java process (running hostpot JVM)
- Find jstack from the same JVM as that of the running process.
- run jstack
The first command line parameter is the process id of the java process to monitor,
after the first option come additional options passed as is to jstack (optional)
</code></pre></div></div>
<h3 id="script-that-continuously-monitors-a-given-java-process">Script that continuously monitors a given java process</h3>
<p>A similar script <a href="https://github.com/MoserMichael/cstuff/blob/master/scripts/jstack/do-jstack-loop">runs jstack at fixed time intervals</a> , this can be used to track locking problems, etc. etc.</p>
<p>It also prints the CPU usage of each java thread, the only other program that knows how to display the CPU usage along the stack trace is “visual vm”: http://visualvm.java.net/
This feature is very useful if you want to detect threads that got stuck in infinite loops for example, those will hang around the same location in the code, but will consume high percentage of CPU time.</p>
<p>It also shows Virtual Memory size + Resident set size for the whole process.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Usage: do-jstack-loop -p <pid> [ -d <delay time> ] [ -o <log file> ]
Description
The script runs jstack in a loop with a period of <delay time> (default value 10 seconds).
Usage:
-p <pid> process to monitor, the process should host the hotspot JVM.
-d <delay> repeat jstack with this delay (in seconds)
-o <logfile> Logs all output to the <logfile>
The script also performs the following steps
- check is <pid> is that of a running process
- check of <pid> is running from the same user account as current account
- check is <pid> is that of a java process (running hostpot JVM)
- Find jstack from the same JVM as from running process.
- run jstack with that version of jvm.
For each JAVA thread it also shows the Operating system thread id (LWP id) and CPU Usage (in %) of that thread.
This information can be used to debug applications that got into infinite loops.
</code></pre></div></div>
<p>The report looks as follows</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>------------------------------------------------------------------------------
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
mmoser 14431 2014 32 70914 11468 0 08:51 pts/0 00:01:54 java -cp . test
2011-08-29 08:57:44
Full thread dump Java HotSpot(TM) Client VM (20.1-b02 mixed mode, sharing):
=== OS Thread Id: 14449 CPU Usage: 3% ===
"Thread-9" prio=10 tid=0xb4636c00 nid=0x3871 waiting on condition [0xb3fd2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at test$1.run(test.java:19)
Locked ownable synchronizers:
- None
=== OS Thread Id: 14448 CPU Usage: 3% ===
"Thread-8" prio=10 tid=0xb4635400 nid=0x3870 waiting on condition [0xb4024000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at test$1.run(test.java:19)
Locked ownable synchronizers:
- None
------------------------------------------------------------------------------
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
mmoser 14431 2014 32 70914 11468 0 08:51 pts/0 00:01:55 java -cp . test
2011-08-29 08:57:46
Full thread dump Java HotSpot(TM) Client VM (20.1-b02 mixed mode, sharing):
=== OS Thread Id: 15241 CPU Usage: 0% ===
"Attach Listener" daemon prio=10 tid=0x08bc3c00 nid=0x3b89 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
</code></pre></div></div>
<h3 id="log-useful-data-for-any-process-in-loop-not-just-java">Log useful data for any process in loop (not just java)</h3>
<p>The following script loops and prints some information about a given process, the gathered information can help to determine if there is a leak of some sort.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./run_top -p <process id> [ -d <delay> ] [ -o <log file> ]
Usage:
-p <pid> process id to monitor
-d <delay> Optional parameter - delay between measurements in seconds.
The default value is 10 seconds
-o <logfile> write output to log file.
Monitors a process continuously and prints the following statistics into a file
- Size of resident set size, i.e. the size of physical memory in use
- Size of Virtual memory, i.e. the size of address space reserved for use
- Number of threads
- Number of sockets/files/pipes.
</code></pre></div></div>
<p>The report looks as follows:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>29/8/2011 9:32:17
Resident set size: 11380 kB
Virtual memory size: 281232 kB
Number of threads: 13
Files: 4
Sockets: 1
pipes: 0
29/8/2011 9:32:27
Resident set size: 11380 kB
Virtual memory size: 281232 kB
Number of threads: 13
Files: 4
Sockets: 1
pipes: 0
29/8/2011 9:32:37
Resident set size: 11380 kB
Virtual memory size: 281232 kB
Number of threads: 13
Files: 4
Sockets: 1
pipes: 0
29/8/2011 9:32:47
Resident set size: 11380 kB
Virtual memory size: 281232 kB
Number of threads: 13
Files: 4
Sockets: 1
pipes: 0
</code></pre></div></div>
Replay UDP packets from capture file.
2011-08-28T00:00:00+00:00
http://mosermichael.github.com//all/projects/2011/08/28/udpsim
["", "projects"] | <h2 id="replay-udp-packets-from-capture-file">Replay UDP packets from capture file.</h2>
<p class="publish_date">
28 August 2011
</p>
<p>Two programs are presented here, one that takes UDP packets from a network capture file and sends them out to a live server, the capture file produced by either tcpdump or wireshark.
The second program takes a capture file and prints out all UDP packets with their payload data into a text file; (I could not persuade the otherwise excellent tshark program to print out the payload data, so I did my own one).</p>
<h3 id="copyright-notice">Copyright notice</h3>
<p>I did this on company time, so the source code of these two programs (<a href="https://github.com/MoserMichael/cstuff/tree/master/iptools">all sources in this directory</a>) is copyright of NDS Group, it is licensed by BSD license. Thanks for my managers for allowing me to release this program as an open source project.</p>
<h3 id="download-instructions">Download instructions</h3>
<p>Get <a href="https://github.com/MoserMichael/cstuff">my stuff</a>
chdir build
./build-iptools.sh
make install
the preceeding line will install to /usr/local
make install INSTALL_PREFIX=/opt/iptools
will install stuff to prefix /opt/iptools</p>
<h3 id="replay-udp-packets-with-udpsim">Replay UDP packets with udpsim</h3>
<p>Basically the simulator performs the following steps</p>
<ul>
<li>G iven a capture file, it loops over all packets included in capture file and selects a subset of packets to be sent out. If the capture file includes IP fragments, these are reassembled into whole packets (if option -n is not set).</li>
<li>The values of IP header and UDP headers are modified in order to fit the current host / target host.
IP and UDP header checksums are recomputed.</li>
<li>Some attempt is made to check if outgoing packet does not collide with any real application running right now (this is not guaranteed to succeed though).
The packets are sent out with the same timing as in capture file; the packet is delayed so that will be sent out at the same time, relative to the start of the capture, this is done by use of high-resolution sleep - actually high-resolution timers work amazingly well.</li>
<li>The packet is sent out to the target host, if the packet is larger than MTU of outgoing interface, it is fragmented (by hand ;-)</li>
</ul>
<p>The simulator also captures network traffic while the simulation is running; by default it is capturing all packets that go via the outgoing network interface.</p>
<p>The program has to be run with root / privileged access; it runs on Linux and worked on Windows under Cygwin (wonders of wonders).</p>
<p>Here is the documentation</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>udpsim [-d] -x <capture_file> -c <capture-host> -i <tx-interface>
[-p <tx-substitution-port>] [-r <tx-substitution-port-range>]
[-f <outgoing-filter-expression>] [-n] [-m]
Resends outgoing UDP traffic from capture file.
Only traffic that originates from one IP address in the capture can be
sent out by this tool, so that we will not be able to
collide with real traffic on the current network segment.
The simulator performs the following steps:
- The simulator loops over all packet contained in the input capture file
- if the source ip address of a packet is equal to <capture-host> value,
then it is selected. Only selected packets
will be sent by the simulator (Please see -c <capture-host> parameter)
- If a filter expression (-f option) is specified then only those packets
from the capture that satisfy the filter are selected.
- All selected packets are subjected to optional substitution rules (-r option);
The first mandatory substitution rule adjusts the source ip of outgoing
packet to that of an existing network interface (-i <tx-interface> options)
Substituion rules specified by -r <rule_syntax> option can modify
the destion port and destination ip address of outgoing packets.
- An effort is made to send the packets with the same timing as those
included in the capture is made.
- During the simulation run we capture all packets that conform to the
filter expression; this way the response is recorded.
An error is reported if source port of selected packet collides with already
bound port.
Arguments:
The substitution rule file includes the following directives:
-x <capture_file>
the capture file in libpcap format (written by tcpdump).
This parameter is mandatory.
-c <capture_host>
Only packets contained in the capture, where source ip address
is equal to <capture_host> will be sent out This parameter is mandatory.
-i <tx-interface>
IP address of a network interface on the current system.
When sending out a replayed packet, the <capture-host> is substituted
with <tx-interface> This parameter is mandatory.
-f <outgoing-filter-expression>
Filter expresssion that is applied to all packets on this capture;
only packets that fit this filter will be sent out.
This parameter is optional.
For syntax of filter expression see see man pcap-filter(7)
-r [src-port|dst-port],<capture-port>,<real-port>
This option specifies an port substituion rule for a an outgoing packet.
This parameter applies either to the destination port (if first token is dst)
or the source port (if first token is src)
When sending a packet, the <capture-port> port number from the capture
file will be substituted with value <real-port>
This parameter is optional, multiple occurences of this option add up.
-r [src-port-range|dst-port-range],<capture-port-start>,<capture-port-end>,<real-port-start>
This option specifies an port substitution rule for outgoing packets.
This parameter applies either to the destination port (if first token is dst)
or the source port (if first token is src)
When sending a packet, the range of port numbers starting with <capture-port-start>
and ending with <capture-port-end>
from the capture file will be substituted with value
<real-port>+<port in packet from capture>-<capture-port-start>
This parameter is optional, multiple occurrences of this option add up.
-r dst-ip,<destination-ip-address-in-capture>,<real-destination-ip-address>
This option specifies an port ip address substitution rule for outgoing packets.
The destination ip address of
-o <output capture file>
This parameter is optional.
File name of capture file of incoming packets.
If this option is missing then the capture file will be stored in a file name
derived from input file name as <input_capture_file>.in.cap
-y <input capture filter>
This parameter is optional.
An expression that is added to capture filter of incoming packets.
By default the capture filter of incoming packets is udp and dst <tx-interface>.
If this parameter is present then its value is added to the default capture filter
and it is set to udp and <input capture filter>
For syntax of filter expression see see man pcap-filter(7)
-n
don't reassemble IP fragments, (default is to reassemble them).
Setting this option means that the MTU of the outgoing interface is
not smaller than the MTU of the capture interface, this is relevant
if the capture file includes fragments, or outgoing packet will be
larger then tx interface MTU.
-m
No packet capture is performed during this run.
-d
Dry run. No packets are sent during dry run (enabled with -d option);
No capture files are created during dry run. Packets data is displayed if
it is sent during a simulation run. If the source port of an outgoing packet
collides with an UDP port in use, then this command exits with error (status 1).
Please note that the list of UDP ports that are used at the moment of
invoking this command is considered.
Limitations - Linux:
user of this program must have root access.
The program tests if outgoing packets would clash with a bound port, alas
list of ports is obtained at start of program run,
this information may not be up to date during the entire simulation run.
The tool gives the possibility to manipulate the ip addresses and port numbers of
outgoing packets, relative to that of the capture.
Not all application protocols may allow for such treatment; if your application
protocol creates a checksum over the ip or udp headers,
then modification of these parameters will break the checksum
(for example IPSEC like protocol over UDP would break).
On some kernel versions (2.6.18) you will have to run 32 bit version of udpsim
on the 32 bit kernel, and 64 bit version on 64 bit kernel.
This is since tpacket_hdr structure (from linux/if_packet.h) is shared between
kernel and user space and has long in it.
When in doubt then please rebuild libpcap and udpsim from the sources.
Limitations - Windows:
Cygwin must be installed (download here http://cygwin.org/ ) and
cygwin/bin directory must be in the path.
Winpcap must be installed (download here http://www.winpcap.org/ ) -
it is also installed as part of wireshark setup.
We can't capture traffic on loopback adapter (winpcap limitation)
General limitations
Currently filters are applied before reassembly of packets, so see section on IP fragments.
Ups this is probably a bug that has to be fixed.
Example usage:
./udpsim -x TEST10_HOST-B.cap -c 10.36.4.112 -i 127.0.0.1 -f 'dst 10.36.4.161' -r dst-port,2000,12321 -r dst-ip,10.36.4.161,127.0.0.1
Uses capture file TEST10_HOST-B.cap (option: -x TEST10_HOST-B.cap)
The following packets are selected to be sent, (all restrictions do apply).
Packet must be udp packet.
the source ip address must be 10.36.4.112 (option: -c 10.36.4.112)
the destination ip address must be 10.36.4.161 (option: -f 'dst 10.36.4.161')
The selected packets are then modified.
The source ip address is set to 127.0.0.1 (option: -f 127.0.0.1)
The destination port is changed from 2000 to 12321 (option: -r dst-port,2000,12321)
The destination address is changd to 127.0.0.1 (option: -r dst-ip,10.36.4.161,127.0.0.1)
Incoming packets are captured by the following default filter: udp and dst 127.0.0.1
Incoming packets are captured in file TEST10_HOST-B.cap.in.cap
./udpsim -x TEST10_HOST-B.cap -c 10.36.4.112 -i 127.0.0.1 -f 'dst 10.36.4.161' -r dst-port,2000,12321 -r dst-ip,10.36.4.161,127.0.0.1 -o in.cap -y "port 12321"
Differences to the previous case:
Incoming packets are captured by the following default filter: udp and dst 127.0.0.1 and port 12321
Incoming packets are captured in file in.cap
</code></pre></div></div>
<h3 id="how-does-udpsim-work">How does udpsim work.</h3>
<ul>
<li>
<p>Packet capture file is read by libpcap</p>
</li>
<li>The packets are sent via RAW sockets (that’s why I have to do defragmentation of packet all on my own ;-)</li>
<li>For capturing packets again libpcap is used (it does not fork out to tcpdump)</li>
</ul>
<p>The program builds its own version of LIBPCAP build, the current release version 1.1.1 had some issues that had to be patched locally.</p>
<h3 id="known-issues">Known issues</h3>
<p>On Linux one should use the 32 bit build with 32 bit kernels, and 64 bit build with 64 bit kernels.
This is due to a beautiful libpcap issue - packet capture and filtering is running in kernel mode, and the result is passed to user mode via a memory mapped file (zero copy indeed). Now the structure had a long member, which ties it to the current built of the operating system, thankfully this issue was removed in recent versions of the kernel; so checking for this issue would make a total mess.</p>
<p>Windows has the issue that one cannot capture packets via the local interface, only real network interfaces do work.</p>
<h3 id="display-packets-from-capture-file">Display packets from capture file</h3>
<p>This is program takes a capture file and prints out all UDP packets with their payload data into a text file; (I could not persuade the otherwise excellent tshark program to print out the payload data, so I did my own one).
It is priceless if you need to write your own quick traffic analysis script in PERL.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>loop_udp -i <infile> [-s] [-f <filter>] [-n]
Print specific information on all udp packets in input capture file.
It shows the following information
seq(#) - sequence number
time(sec.usec) - time of packet
ipv4_src - source ip
udp_port - udp port
ipv4_dst - destination ip
udp_dst - destination port
data_len - length of UDP payload
data - hexadecimal dump of UDP payload.
By default, IP fragments are reassembled into packets, this requires
that fragments have been captured and are present in the capture file.
For whatever reason I could not bring tshark to show packet data,
so i did my own, by using libpcap;
-i <infile> - input capture file
-s - don't show header line
-f <filter> - filter expression (for syntax see man pcap-filter(7))
-n - don't reassemble IP fragments, (default is to reassemble them)
</code></pre></div></div>
<h3 id="ip-fragmentation">IP Fragmentation</h3>
<p>An IP packet that is too big will be <a href="http://en.wikipedia.org/wiki/IPv4#Fragmentation_and_reassembly">divided into fragments</a> . Fragmentation starts when a packet is sent that is larger than the network cards MTU - maximum transmission unit, on Ethernet adapters this value is typically 1500 bytes;</p>
<p>One point to remember when the network capture is taken:
One might be dealing with packets that can get larger than the interface MTU, in this case we would like to capture all fragments of an IP packet - all the pieces that make up the packet, so the capture filter should only be relative to IP information (i.e. source IP address and destination IP address);</p>
<p>Why? The capture filter is applied separately on each packet, just when the packet is arriving at the network card; so an IP fragment arrives without the UDP header, the UDP header is only included with the first fragment.
Therefore a capture filter with the UDP port will just ignore an IP fragment that comes after the first fragment, as the UDP port number cannot be checked without the UDP header.
Of course a network capture can grow insanely large, when all packets that go through it are captured.</p>
<p>Why can’t libpcap reassemble fragments? Libpcap network capturing is done by the kernel, so when you specify a capture filter in tcpdump, then tcpdump compiles the filter into a finite state machine that implements the capture filter; this compiled state machine is passed to the kernel and the kernel applies it to each incoming packet (for packets that go through the specified capture device). Now each packet that is filtered, goes into a circular buffer structure that is placed into a shared memory segment and can be seen directly in user space code; where tcpdump picks it up and places it into the capture file. Actually this is a form of <a href="http://en.wikipedia.org/wiki/Zero-copy">zero copy</a> networking, which is a form of Rocket science for networking; by mapping the data through share memory, the packet does not have to be copied back and forth from kernel address space to user address space on gazillion system calls (what you do have when calling recv and send though).
Now of course memory is premium in kernel space, and the kernel component responsible for capturing can’t be bothered with IP reassembly.</p>
<p>Well, at least the TCPDUMP user space could be bothered with IP reassembly? Probably not, for gigabit network cards you would pass enormous amounts of data back and forth, so doing IP reassembly in user space on the fly would be a stretch. Well it might be possible but.</p>
<p>… but actually most applications that use UDP should very much avoid IP fragmentation, because it puts a large strain on the system, when packets are broken up on one side, and reassembled on the other side. Of course a fragment might be lost, in this case it has to be resent, so forth. IP fragmentation is generally a thing that an application would like to avoid.</p>
<p>With TCP/IP - IPv4 - there might be a case when IP fragmentation occurs; if the packet goes through multiple hops between routers, and two hops have a MTU that is smaller than the size of a packet. In IPv4 the router would do fragmentation on its own. Now luckily this situation is generally avoided - there is the magical size of 576 bytes, where IP packets are guaranteed to pass without fragmentation. (or supposed to be so).</p>
<p>So fragmentation is sort of an edge case that all systems try to avoid. All systems ? Well there is no such thing, always some exceptions exist ;-)</p>
Script to set up SSH trusted keys
2011-07-14T00:00:00+00:00
http://mosermichael.github.com//all/projects/2011/07/14/ssh-friends
["", "projects"] | <h2 id="script-to-set-up-ssh-trusted-keys">Script to set up SSH trusted keys</h2>
<p class="publish_date">
20 July 2011
</p>
<p>If you ever have to type the same sequence of three commands again, then turn this into a script - that’s a general rule on UNIX, or any other systems for that matter were the command line rules.</p>
<p>If you have to run process or copy files to another machine, than you are probably working with <a href="http://www.openssh.com/">OpenSSH</a> . One can establish a trust relationship between computer A / account A and Computer B / account B, so that ssh will not ask for a password when logging on to Computer B /account B while you are on computer A / account.</p>
<p>This effect is achieved by means of sharing a public / private key pair - hence the name trusted key; Computer A owns the private key, computer B receives the public key; During the logon process, computer A encrypts a message with its private key (only computer A is supposed to have this private key); computer B can use it’s public key in order to decrypt and verity the message, so that computer B knows that its computer A who is calling, and not some imposter.</p>
<p>Unfortunately it takes quite a long sequence of commands to set this up correctly, also most of the guides on the Internet that describe the process get it wrong. So it’s time for some scripting magic, the script will automate set up of trusted key. First of all the script can be <a href="https://github.com/MoserMichael/cstuff/blob/master/scripts/maketrust.sh">seen here</a></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./maketrust.sh -m <hostname> -u <user> [ -p <password> | -P | -d | -h]
This script automates setup of trusted key between the currently logged on
user account on the current host on the one hand, and the
remote <hostname> and <user> account on the other hand.
If the command succeeds then SSH/SCP/SFTP commands will not require
a password when logging on to <user>@<hostname>, while your are logged
in on the current user on current host.
Command line options
-m Remote host name
-u Remote user
-p password of remote machine (-P option is preferable, if you do not want others to
see your password with ps command, while this script is running).
-P prompt the user to enter command via terminal.
-d (optional) debug option. Creates log files maketrust.log and maketrust.log.expect.
</code></pre></div></div>
<p>So the process is as follows. Lets say
o $HOSTNAME and $USERNAME are the local host and user names.
o $RHOSTNAME and $RUSERNAME are the remote host and user names.</p>
<p>* On the local computer the public and private key pair is created</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ssh-keygen -t dsa -f id_dsa -P ''"
</code></pre></div></div>
<p>* On the local computer installs the public and private key pair</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkdir -p ~/.ssh/${USERNAME }@${ HOSTNAME}/${RUSERNAME}@${RHOSTNAME}
mv id_dsa ~/.ssh/${USERNAME }@${ HOSTNAME}/${RUSERNAME}@${RHOSTNAME}
mv id_dsa.pub ~/.ssh/${USERNAME }@${ HOSTNAME}/${RUSERNAME}@${RHOSTNAME}
</code></pre></div></div>
<p>* On the local computer: edit the local machine’s SSH configuration so that it
will pick up the correct public/private key pair.
Add the following line to ~/.ssh/config file</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>IdentityFile ~/.ssh/ident/%u@%l/%r@%h/id_dsa
</code></pre></div></div>
<p>Now the last two steps were very important: SSH will now look for the keys in specific location, and you can install multiple keys without them interfering with one another; This IndentityFile definition will instruct OpenSSH to look for each trusted key in its separate directory: <local user="" name="">`<local host="" name="">/<remote user="" name="">`<remote host="" name=""> ; the local host and user name is used here because the home directory may be mounted on an NFS disk, so the same home directory can be used while logging on from different systems (in the general case).
Correction: this trick works only from version 4.4 of OpenSSH on; on previous versions (that is beginning with version 2.5.1.p1 until version 4.4) all you can do is add multiple IdentityFile entries, one for each private key owned by computer A. For these older versions we can have multiple definitions of IdentityFile directive, each one lists a separate trusted key. Now the OpenSSH client will try out each trusted key in sequence, until it either finds a matching key or gives up. Interestingly Redhat keeps old versions of the OpenSSH package even in recent releases; maybe this policy is supposed to mirror that of the Debian stable release - but what am I to know about 'Enterprise software' in general.
Kindly the [setup script](https://github.com/MoserMichael/cstuff/blob/master/scripts/maketrust.sh) takes care of all these details.</remote></remote></local></local></p>
<p>* Set correct permissions (only the current user should be allowed to read from .ssh)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>chmod 0700 ~/.ssh
chmod 0700 ~/.ssh/${USERNAME }@${ HOSTNAME}
chmod 0700 ~/.ssh/${USERNAME }@${ HOSTNAME}/${RUSERNAME}@${RHOSTNAME}
chmod 0700 ~/.ssh/${USERNAME }@${ HOSTNAME}/${RUSERNAME}@${RHOSTNAME}
chmod 0600 ~/.ssh/config
chmod 0600 ~/.ssh/${USERNAME }@${ HOSTNAME}/${RUSERNAME}@${RHOSTNAME}/id_dsa
chmod 0600 ~/.ssh/${USERNAME }@${ HOSTNAME}/${RUSERNAME}@${RHOSTNAME}/id_dsa.pub
</code></pre></div></div>
<p>* On the remote machine
Copy id_dsa.pub to the remote machine
Append content of id_dsa.pub to ~/.ssh/authorized_keys2</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat id_dsa.pub >>~/.ssh/authorized_keys2
</code></pre></div></div>
<p>Create the symbolic link</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ln -s ~/.ssh/authorized_keys2 ~/.ssh/authorized_keys
</code></pre></div></div>
<p>Again set all permissions</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>chmod 0700 ~/.ssh
chmod 0600 ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/ authorized_keys2
</code></pre></div></div>
<h3 id="scripting-magic">Scripting magic</h3>
<p>Basically it’s a shell script generates the keys and installs them on local computer, then it that writes an installation shell script that contains the public key, later it writes two expect script to copy & run the installation script on another machine. Sort of <a href="http://dictionary.die.net/toolsmith">I’d rather write programs to write programs than write programs</a>
By making a script out the long wound process, I would see the error that is apparently hidden in others tutorials that cover this subject.</p>
<h3 id="portability-of-shell-scripting">Portability of shell scripting</h3>
<p>Once upon a time you could rest assured that each Linux installation has a bash script shell; now that is no longer true. Ubuntu for instance gave up bash in favor of the dash shell.
Well, I am still sticking with bash though; but any software sort of dies due to sudden changes of its environment; I guess this will true as long as you do not have <a href="http://en.wikipedia.org/wiki/WALL-E">real talking and improvising robots</a> (Also see my <a href="http://mosermichael.github.com/cstuff/all/ramblings/2011/06/18/semantic-nets.html#we">Wall-E test of intelligent systems</a> ).</p>
My take on software (Re)use
2011-07-11T00:00:00+00:00
http://mosermichael.github.com//all/blog/2011/07/11/documentation
["", "blog"] | <h2 id="my-take-on-software-reuse">My take on software (Re)use</h2>
<p class="publish_date">
11 August 2011
</p>
<p><a href="http://en.wikipedia.org/wiki/Code_reuse">Code reuse</a> is like a <a href="http://en.wikipedia.org/wiki/Holy_Grail">Holy Grail</a> of sorts for software engineering; Magical Powers of productivity are attributed to code reuse, however the concept has proven to be somewhat illusive in practice. Most attempts to achieve this aim also tend to lead to doubts and frustration. Yosefk has <a href="http://www.yosefk.com/blog/redundancy-vs-dependencies-which-is-worse.html">a long and thorough piece about that</a> - the argument here is that by using a library one adds dependencies, managing dependencies adds overall complexity (paradoxically) - this complexity is created by the need of managing those dependencies, requires one to adapt to the libraries interface, creates politics from the need to interact with the author / origin of the said library etc. etc. As usual with software there are tradeoffs to make; as usual the resulting choice is often determined by the character of the person that makes it ;-)</p>
<p>By the way, the stuff here for <a href="https://github.com/MoserMichael/cstuff">my projects</a> that might be usable to others is document <a href="http://mosermichael.github.com/cstuff/docs.html">right here</a> .</p>
<p>I would like it simpler, so here is my tale: Once upon a time I have worked as a free lance programmer/consultant; the problem here is that the software consultant (me that is) had to determine up front the exact amount of work required for each job. My customers somehow always needed to know how long it takes / how much money they would have to pay me. So after one or two miscalculations I have come up with the following rules.</p>
<ul>
<li>Each entity of work basically needs at least three iterations</li>
<li>First iteration ‘does something’; it basically works according to how I understand the problem. The main mistake in consulting is to mistake the result of first iteration for the end result.</li>
<li>The second iteration comes nearer</li>
<li>The third iteration must then be according to customer expectations</li>
</ul>
<p>Basically this knowledge translates into the rule of thumb where everything takes at least three times as longer than expected.
So in my definition a software component is</p>
<ul>
<li>A software entity that has some defined boundary</li>
<li>It requires at least three iterations to get it reasonably right.</li>
<li>You have a 95% chance of getting it wrong the first time when you do all by yourself</li>
</ul>
<p>This definition is somewhat vague; however it should be applied as a test to see if there is a justification for creating or using another component. If you see an entity in your ‘problem domain’ and are certain that these three rules are true, then a programmer should consider to look out if there is a software component already written by somebody else that covers the ‘problem domain’.</p>
<p>P.S. If the reader of this text somehow happen to be a manager of sorts, then please be aware that assigning junior people to reusable stuff is a sure receipt for disaster. This approach can be characterized by the phrase ‘Infrastructure is not a real project’ which is a quite funny phrase if you think of it.</p>
Stuff that we do with computers
2011-07-04T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2011/07/04/pride
["", "ramblings"] | <h3 id="i-did-it">I did it!</h3>
<p>An enormous feeling of accomplishment grips me - I changed the ringtone on my cell phone.</p>
<p>First thing is to find some piece of music on <a href="http://youtube.com">youtube</a> that would fit the occasion.</p>
<p>Then I had to figure out how to download it (One option is to use the <a href="https://addons.mozilla.org/en-US/firefox/addon/fast-video-download-with-searc/">Fast Video Download plugin</a> for <a href="http://www.mozilla.com/">firefox</a> (actually the wide range of plug-ins keeps me from switching to <a href="http://www.google.com/chrome/">Chrome</a> browser )</p>
<p>Now the downloaded result is in flv format; Next thing is to convert the resulting flv file to mp3 formatby using <a href="http://www.ffmpeg.org/">ffmpeg</a> converter. The following command does it (good old command line to the rescue)
ffmpeg -I <flv file="" name=""> out.mp3</flv></p>
<p>Then one has to can edit the mp3 file with the “audacity”: http://audacity.sourceforge.net/ sound editor, and cut out the portion of the tune that will be the ringtone. Now the result can only be saved to mp3 format after downloading an additional of <a href="http://lame.buanzo.com.ar/">lame converter for audacity</a></p>
<p>Next thing is an enormously complex operation that one has to do in order hook up Windows with the Bluetooth device on my cell phone;</p>
<p>*First Windows has to see the Bluetooth device; luckily the laptop comes with a Bluetooth radio; In windows 7 control panel search for ‘Add a Bluetooth device’ and add your cell phone;</p>
<ul>
<li>Pair the laptop with the phone (On laptop screen a number appears that has to be punched into the cell phone)</li>
<li>The directory where you copy the file has to be made ‘visible via bluetooth’</li>
<li>One the Laptop search for the “Bluetooth file transfer wizard” to the cell phone. Select file to copy and wait (I can’t describe all the detail, but in the end it did work probably after turning on this option and that option on the phone, pure magic).</li>
</ul>
<p>Next is to find the file on the cell phone’s file system; then I had to find a hidden option in an extremely confusing menu system that would turn a mp3 file in to a ring tone.</p>
<p>Now one general question remains though; why is every little thing you want to do with a computer turning into a seven stage process?</p>
<p>That reminds me of the lamentations from the <a href="http://www.net-snmp.org/docs/FAQ.html#What_is_the_Official_Slogan_of_the_net_snmp_coders_list_">netsnmp FAQ</a> which says</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>What is the Official Slogan of the net-snmp-coders list?
-------------------------------------------------------
"The current implementation is non-obvious and may need to be improved."
(with thanks to Rohit Dube)
And an alternate, added 26-Apr-2000:
"In theory, it shouldn't be that hard, but it just needs to be done."
</code></pre></div></div>
<p>Another point is this
The Song <a href="http://www.youtube.com/embed/3C11AlNqaOo">Wenn der Topf aber nun ein Loch hat</a> in German</p>
<iframe width="425" height="349" src="http://www.youtube.com/embed/3C11AlNqaOo" frameborder="0" allowfullscreen=""></iframe>
<p>The English version <a href="http://www.youtube.com/embed/rBHdZj-qkeQ">There’s a whole in the Bucket</a></p>
<iframe width="560" height="349" src="http://www.youtube.com/embed/rBHdZj-qkeQ" frameborder="0" allowfullscreen=""></iframe>
<p>Now this song is supposed to introduce kids to long-wound and tedious industrial processes. Isn’t it? Nursery rhymes as the foundation of the wealth of nations! Applause, applause for this genuine contribution!!! Well, at least I can prove that my claims are ridiculous - as nothing suspecting little kid certainly did not understand the coded message in these songs, namely that I am up for a lot of tedious work going through the process of fixing and fixings pots that is (or maybe they are actually honey jars, but that would be a different story).</p>
<p>Of course playing with computers all day is so much cooler than standing all day in front of an assembly line.</p>
<iframe width="425" height="349" src="http://www.youtube.com/embed/MHdmaFJ6W6M" frameborder="0" allowfullscreen=""></iframe>
<p>(Whoa, They got the whole Film ‘Modern Times’ by Chaplin Online ! .. And the Great Dictator is <a href="http://www.youtube.com/watch?v=8Tz0gGgpSM8&feature=related">online too</a> !!! )</p>
<p>Now I should really be thankful for living in post-modern times; even if these post-modern times come with a lot of post-modern bla-bla, and post-modern ga-ga.</p>
<p>Correction, even <a href="http://www.prospectmagazine.co.uk/2011/07/postmodernism-is-dead-va-exhibition-age-of-authenticism/">postmodernism is dead</a> now, they are doing retrospective exhibitions about it ;-)</p>
Scrum and what makes organizations tick
2011-06-20T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2011/06/20/organizations
["", "ramblings"] | <h2 id="scrum-and-what-makes-organizations-tick">Scrum and what makes organizations tick</h2>
<p class="publish_date">
20 July 2011
</p>
<p>Organizations all have to solve a common problem; they somehow have to motivate their workers to do their best and to toe the line. Especially big organization have to cope with the forces of <a href="http://en.wikipedia.org/wiki/Marx%27s_theory_of_alienation">alienation / enstrangement</a> of an abstract workforce; everybody is doing his very tiny part of the whole, and certainly most people do not have a global picture of what is happening / do not care about it at all. There are all sorts of <a href="http://www.monroecc.edu/wusers/mmarriott/bus204/ch13.htm">psychological models</a> of how to motivate a ‘workforce’ ; mostly it tries to identify peoples ‘needs’ versus ‘expectations’ whatever / in a way it’s like most psychology - about how to press the right button / stimulus - response that is.
I think that there can be the following basic approaches:</p>
<ul>
<li>The big carrot trick; Once upon a time, with fast growing companies you had the incentive of vesting stock options; now that is somehow less tangible, except for a few companies that are also very picky with their hiring process ;-)</li>
</ul>
<!-- -->
<ul>
<li>Building of <a href="http://en.wikipedia.org/wiki/Pecking_order">pecking orders_</a> - hierarchy building. Every member of the organization is supposed to value his position in the order of things and strive to get to a higher rung. Also there is the very personal question of ‘where will I be once I leave the current pecking order’ that is supposed to
be a factor of employee retention. However the end result of this effort often stinks.</li>
</ul>
<!-- -->
<ul>
<li>Another lever is the degree of autonomy that a lower level manager gets. However I do think that a high degree of autonomy breeds serfdom at the workplace, power seldom brings out the best in man, especially not among control oriented geeks.</li>
</ul>
<!-- -->
<ul>
<li>Exploration into <a href="http://www.esrad.org.uk/resources/vsmg_3/screen.php?page=introduction">Non hierarchical methods</a> of building and organization. For example the Viable Systems model tries to <a href="http://www.esrad.org.uk/resources/vsmg_3/screen.php?page=1qguide">view an organization as a system model</a> that stresses the importance of such concepts as Information flow / feedback between the different parts of an organization. Optimization of information flow is regarded as a major task, which also requires total transparency and also everybody needs to have a say in things - participation that is.</li>
</ul>
<p>Very interesting approach, unfortunately <a href="http://en.wikipedia.org/wiki/Anthony_Stafford_Beer">Professor Stafford Beer</a> , the author of this approach was an <a href="http://en.wikipedia.org/wiki/Project_Cybersyn">management consultant / adviser for the Chilean government of Salvador Allende</a> which sort of labeled him in terms of ideology. I think it’s a pity, because the cybernetic approach of studying organizations is not a question of who owns what, it’s a question of how to run a business.</p>
<p>Fast forward to the WEB 2.0 world where we now have the <a href="http://en.wikipedia.org/wiki/Scrum_%28development%29">Scrum methodology of (software) development</a>
The aim of Scrum takes is to build <a href="http://stackoverflow.com/questions/4231226/what-does-self-organizing-scrum-team-mean">Self Organizing teams</a> - that is nobody can tell the team the details of how to create the product (sort of the Manager is not supposed to interfere with the technical details). Also a large part of the process is about information flow and how to incorporate early feedback into the (software) product.</p>
<p>(Personally I dislike Scrum stand up meetings, as they are always held early in the day - the time of day where I tend to do things productively) However the concept is certainly an interesting one … Another problem is that Scrum started out as a simple methodology, but then started to be ever more complex; (this is the way of things with Software). So with a sophisticated process you sometimes lose touch even with stated ‘overriding principles’ that is … And that’s probably the way with all Ideologies, in a larger sense.</p>
<p>Another point is that SCRUM when compared with the <a href="http://www.esrad.org.uk/resources/vsmg_3/screen.php?page=introduction">Viable System Model</a> has much less rigor. SCRUM, unlike the “Viable System Model”, is not a discipline of analyzing systems or processes; it is more of a set of procedures that promise to set up an environment in order to enable people to ‘get things done’ (sounds very amorphous- they like that with software).</p>
<p>Also interesting to note is that the <a href="http://en.wikipedia.org/wiki/Scrum_%28development%29">Scrum methodology of (software) development</a> people don’t know anything about this <a href="http://www.esrad.org.uk/resources/vsmg_3/screen.php?page=introduction">Viable System Model</a> , though both approaches do share a lot of ideas …</p>
<p>(Current Music: <a href="http://www.youtube.com/watch?v=1awtlA6Nr64">Crossby, Stills and Nash: You don’t have to cry</a> )</p>
<iframe width="420" height="345" src="http://www.youtube.com/embed/JCqo89Wjh20" frameborder="0" allowfullscreen=""></iframe>
People turn sour when the weather turns bad
2011-06-19T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2011/06/19/climate
["", "ramblings"] | <h2 id="people-turn-sour-when-the-weather-turns-bad">People turn sour when the weather turns bad</h2>
<p class="publish_date">
19 July 2011
</p>
<p>A <a href="http://en.wikipedia.org/wiki/4.2_kiloyear_event">hundred year drought</a> did occur somewhere around 2.200 BC, and as the weather turned nasty, so did politics. Both the Old Egyptian Empire and Akkadian Empires collapsed, there was no centralized government at all. And so it goes that civilization took a break for two hundred years (the institution of civilization was closed for repairs).</p>
<p>Well, I don’t quite know about the extent of that; somebody must have stayed in charge, at least locally. Ancient civilizations were dependent on their <a href="http://mygeologypage.ucdavis.edu/cowen/~GEL115/115CH17oldirrigation.html">Irrigation projects</a> ; these systems have to be maintained constantly, otherwise they sustain irreparable damage . Anyway, it was <a href="http://en.wikipedia.org/wiki/Ankhtifi">nasty</a> very <a href="http://en.wikipedia.org/wiki/Ipuwer">nasty</a> ; this calamity also known as the <a href="http://en.wikipedia.org/wiki/First_Intermediate_Period_of_Egypt">first intermediate period</a> (nice name).</p>
<p>After this epic collapse, life seems to have been much more pleasant during the Middle Kingdom; One important ideological innovation of this time is Maat - The Order of Things with a capital letter (somehow the concept seems similar to the <a href="http://en.wikipedia.org/wiki/The_Tao_of_Pooh">Tao</a> ; though the concept was present in the Old Kingdom it is said to have gained in importance during the Middle kingdom.</p>
<blockquote>
"In addition to the importance of the Maat, several other principles within Ancient Egyptian law were essential, including an adherence to tradition as opposed to change, the importance of rhetorical skill, and the significance of achieving impartiality, and social justice. In one Middle Kingdom (2062 to c. 1664 BCE) text the Creator declares "I made every man like his fellow". Maat called the rich to help the less fortunate rather than exploit them, echoed in tomb declarations: "I have given bread to the hungry and clothed the naked" and "I was a husband to the widow and father to the orphan".\[10\] "
[Source](http://en.wikipedia.org/wiki/Maat</blockquote>
<p>)</p>
<p>So it seems that after a really bad calamity the political class decides that something profound has to be done in order to strengthen political stability. Like after World War II (4000 years later that is) when there was the wider New Deal/ Social democratic consensus/Soziale Marktwirtschaft system which implied that sustained political stability would require a profound betterment in the conditions of the common man; in conjunction with the <a href="http://en.wikipedia.org/wiki/Bretton_Woods_system">Bretton-Woods system</a> it helped to improve living conditions in the western world at an unprecedented scale; and also real wages did stagnate ever since it was abandoned in the late seventies.</p>
<p>Another point is that as the climate turns bad, things turn nasty and people turn ugly; For example the <a href="http://en.wikipedia.org/wiki/Bronze_Age_collapse">Bronze age collapse</a> is said to be due in part to a worsening climate triggered by a <a href="http://en.wikipedia.org/wiki/Hekla_3_eruption">monumental Volcano eruption</a> ; the Middle ages got really nasty after the onset of the <a href="http://en.wikipedia.org/wiki/Little_Ice_Age">Little Ice Age</a></p>
<p>I wonder what the political implications of worsening economic conditions due to global warming will look like. How will our present day enlightened political culture justify a return to a nasty order of things? I guess they will find an answer, they always do … Kurt Vonnegut said that our society is basically <a href="http://www.inthesetimes.com/article/cold_turkey/">addicted to cheap oil</a> now that period is clearly coming to an end.</p>
<p>How will change come about? Like in the twentieth century, all sort of calamities come togather with rationing; Smart electric grids will enable the powers that be to impose effective rationing of energy resources; this in turn will justify rationing of food and many other things as well. There will be social casts distinguished by differences in the rationing schema, and this in turn will introduce political inequalities.</p>
<p>Change we can believe in.</p>
<p>(Current Music: <a href="http://www.youtube.com/watch?v=ua0pR06pevU">The Great Nations of Europe in the 16th Century</a></p>
<iframe width="425" height="349" src="http://www.youtube.com/embed/ua0pR06pevU" frameborder="0" allowfullscreen=""></iframe>
StackMirror tool for fixing of stack corruption problems
2011-06-19T00:00:00+00:00
http://mosermichael.github.com//all/projects/2011/06/19/stack-mirror
["", "projects"] | <h2 id="stackmirror-tool-for-fixing-of-stack-corruption-problems">StackMirror tool for fixing of stack corruption problems</h2>
<p class="publish_date">
19 July 2011
</p>
<p>Debugging <a href="http://en.wikipedia.org/wiki/Buffer_overflow">buffer overruns</a> is hard. Usually you have to solve these bugs with logs + apply divide and conquer techniques. Even <a href="http://valgrind.org/">valgrind</a> can’t help here !
GCC has the compiler options -fstack-protector-all , but it does not give accurate stack traces, it is more designed to shut down an application so as to avoid <a href="http://insecure.org/stf/smashstack.html">remote exploit</a> of a buffer overflow error.</p>
<p>So a new open source tool is presented here that can help with most such cases; StackMirror to the rescue! It detects even the nastiest stack smashers and reports an accurate stack trace of the function that caused it.</p>
<p>First of all how to use it</p>
<ul>
<li>
<p>The tool works with gcc (versions or upward)</p>
</li>
<li>
<p>The application must be re compiled with gcc compiler option -finstrument-functions</p>
</li>
<li>
<p>During runtime the application has to load stack.so library, so either LD_PRELOAD environment has to be set to the shared library, or application has to link with it.</p>
</li>
<li>Application should not be compiled with Optimization (-O1 -O2 -O3 ) and frame pointer omission (Please use both -fno-omit-frame-pointer and -On in order to prevent frame pointer omission).</li>
<li>I have tested the tool on both i686 and x86_64; please look if it works on other platforms too.</li>
</ul>
<h3 id="download-instructions">Download instructions</h3>
<p>Get <a href="https://github.com/MoserMichael/cstuff">my stuff</a>
chdir stack
make
(for specific platform builds do make ARCH=x86_64 or make ARCH=i686 )
make install
the preceeding line will install to /usr/local
make install INSTALL_PREFIX=/opt/stack
will install stuff to prefix /opt/stack</p>
<h3 id="step-by-step-usage-instruction">Step by step usage instruction</h3>
<p>Prior to running your program, please set the following environment variable
export LD_PRELOAD=/usr/local/lib/libstack.so
Please note that value of LD_PRELOAD must be the full path name of libstack.so</p>
<p>Run you program, if it smashes the stack and we are ok then an error message of the following form is displayed. Don’t worry; it will be formatted in the next step.</p>
<pre>
ERROR: Stack corruption detected
**** start stack ***
frame 0 ip: 0x80484e4
frame 0 ip: 0x80485c3
frame 1 ip: 0x804856f
frame 1 ip: 0x80485e7
frame 2 ip: 0x804856f
frame 2 ip: 0x80485e7
frame 3 ip: 0x804856f
frame 3 ip: 0x80485e7
frame 4 ip: 0x804856f
frame 4 ip: 0x80485e7
frame 5 ip: 0x804856f
frame 5 ip: 0x80485e7
frame 6 ip: 0x804856f
frame 6 ip: 0x80485e7
frame 7 ip: 0x804856f
frame 7 ip: 0x80485e7
frame 8 ip: 0x804856f
frame 8 ip: 0x80485e7
frame 9 ip: 0x804856f
frame 9 ip: 0x80485e7
frame 10 ip: 0x804856f
frame 10 ip: 0x804864e
frame 11 ip: 0x8048627
frame 11 ip: 0x80486c7
frame 12 ip: 0x8048670
frame 12 ip: 0x40049e37
08048000-08049000 00000000 /home/mmoser/archive/cstuff/GNU-Linux-2.6.38-8-generic-i686/test/tst
40000000-4001c000 00000000 /lib/i386-linux-gnu/ld-2.13.so
4001e000-4001f000 00000000 [vdso]
40021000-40023000 00000000 /home/mmoser/archive/cstuff/GNU-Linux-2.6.38-8-generic-i686/lib/libstack.so
40033000-4018d000 00000000 /lib/i386-linux-gnu/libc-2.13.so
*** eof stack ***
</pre>
<p>Save this error message to a file crash.log and run
/usr/local/bin/readerrors.pl crash.log</p>
<p>It should create the following message:</p>
<pre>
-----------------------------
ERROR: Stack corruption detected
-----------------------------
ip 0x80484e4 from /home/mmoser/archive/cstuff/GNU-Linux-2.6.38-8-generic-i686/test/tst
crash_me
test.c:10
ip 0x80485c3 from /home/mmoser/archive/cstuff/GNU-Linux-2.6.38-8-generic-i686/test/tst
fact
test.c:36
ip 0x804856f from /home/mmoser/archive/cstuff/GNU-Linux-2.6.38-8-generic-i686/test/tst
fact
test.c:30
ip 0x80485e7 from /home/mmoser/archive/cstuff/GNU-Linux-2.6.38-8-generic-i686/test/tst
fact
test.c:38
ip 0x804856f from /home/mmoser/archive/cstuff/GNU-Linux-2.6.38-8-generic-i686/test/tst
fact
test.c:30
ip 0x80485e7 from /home/mmoser/archive/cstuff/GNU-Linux-2.6.38-8-generic-i686/test/tst
fact
test.c:38
ip 0x804856f from /home/mmoser/archive/cstuff/GNU-Linux-2.6.38-8-generic-i686/test/tst
fact
test.c:30
ip 0x80485e7 from /home/mmoser/archive/cstuff/GNU-Linux-2.6.38-8-generic-i686/test/tst
fact
test.c:38
</pre>
<h3 id="how-does-it-work">How does it work?</h3>
<p>Compiler option -finstrument-function is usually used for building profilers; for each function the compiler inserts two function calls; on entering the function it calls _<em>cyg_profile_func_enter on leaving the function it calls</em>_cyg_profile_func_exit .
The tool implements these two functions; on entering an application function the tool records the function return address, and when it is time to leave this function it checks if that address has been modified; if that is the case then we know that the stack was smashed here right by this function.</p>
<p>In more detail: For each application thread a stack data structure is maintained and referenced by a thread local storage entry. For each function call this stack keeps an entry; this entry contains the function return address, the function frame pointer address and the value received from dereferencing the frame pointer address. Typically the value obtained from dereferencing of frame pointer address is the return address from the called function. The stack entry is created when _<em>cyg_profile_func_enter is called. When the function is left and</em>_cyg_profile_func_exit is called, we check that the value obtained from dereferencing of the frame pointer did not change. If it actually did change, then we know for certain that the stack was overwritten, just right now here by this function. We also have the accurate application stack trace, just by looking at the entries in the stack data structures. So when an error occurs, the return address of all functions is printed to standard error, in addition to that, we print the addresses of the text segment for all shared libraries + executable. This information is then converted (offline after the crash) to symbolic names by the readerrors.pl script ; we do not want to do this at crash time.</p>
<p>The tool keeps entries that shadow the application stack - hence the name StackMirror. Of course this shadow stack can also be overwritten by some pointer gone wild, but this is much less likely than occurrence of a buffer overflow error. Another limitation is that we only return address of the current function is checked. In some complex example an application function call could smash the return address of a calling function; we do not check for that here, (though an even more paranoid version of the tool could do that). We will still get there eventually, as the stack unwinds.</p>
<p>.h3 Other limitations</p>
<p>This library makes an effort to recover from situations with <a href="http://en.wikipedia.org/wiki/Setjmp.h">setjmp / longjmp</a> trick; this function is a non local form of goto ; it undoes the stack up until some remembered levy.
However there is one trick that we can’t cope with <a href="http://en.wikipedia.org/wiki/Setcontext">makecontext / setcontext</a> If the application decides to do its own cooperative multitasking / user level threading or if it needs co procedures then all our efforts here are toast.
Note that it’s becoming something of a habit here - for every trick find when it breaks ;-)</p>
Semantic nets and social networks
2011-06-18T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2011/06/18/semantic-nets
["", "ramblings"] | <h2 id="semantic-nets-and-social-networks">Semantic nets and social networks</h2>
<p class="publish_date">
18 July 2011
</p>
<p>Man has named child after <a href="http://www.bbc.co.uk/news/world-middle-east-13417930">a Button</a>
(At least it is not <a href="http://en.wikipedia.org/wiki/Big_red_button#Symbolism">this button</a> )</p>
<p>While staying on the subject of Facebook, I wonder if you can automate creation of <a href="http://en.wikipedia.org/wiki/Semantic_network">semantic nets</a>) by doing statistical analysis of things that people do ‘like’. In order to do that you would need information kept by both google and facebook; Facebook knows what you ‘like’ (corresponding to the concepts of a semantic net) Google knows what you have searched for and how you got it (the edges between concepts) / the edges are extracted from query words. Nodes are concepts that you ‘liked’. Ups now google <a href="https://plus.google.com">has that too</a></p>
<p>This sort of research would be similar to what they are doing in Natural Language Processing, these days it’s all statistics and <a href="http://en.wikipedia.org/wiki/N-gram">n-grams</a> With such an encompassing knowledge system the resulting Knowledge Monopoly ™ could serve you content aware adds on future thoughts even before you have thought of them.</p>
<p>.. Now there was this recent science fiction idea of the <a href="http://www.amazon.com/dp/0670033847/">Singularity event</a> being <a href="http://www.fourmilab.ch/fourmilog/archives/2011-02/001293.html">near</a> - It says that a machine capable of thought will blow away all current technological limitations and we will all shoot out for the stars, and all that should happen Really soon.</p>
<p>Well I don’t know, even if the machine comes up with a grand plan, it will depend on the political will of Humans to implement it; so I guess that all the bots will end up sending emails to each other being largely ignored … Just like us Humans ;-) also we might end up with a <a href="http://en.wikipedia.org/wiki/Marvin_the_Paranoid_Android">really depressed Marvin</a></p>
<h3 id="how-to-test-the-intelligence-of-machines">How to test the intelligence of machines</h3>
<p>With the aim of furthering the Singularity event - I would now like to propose a better test that should tell if a machine is intelligent or not.</p>
<p>Normally a program dies when its environment changes in some unexpected ways (this is also called <a href="http://en.wikipedia.org/wiki/Software_rot">Software rot</a> ); I would say that a program (or any other technical system) is intelligent, if it can adapt all by itself to unexpected changes of its environment or adjust to unexpected changes in its input. By that definition a search engines that produces good results for any queries makes an intelligent system.</p>
<p>Interestingly now I remember that the book <a href="http://en.wikipedia.org/wiki/The_Ghost_in_the_Machine">The Ghost in the Machine</a> by <a href="http://en.wikipedia.org/wiki/Arthur_Koestler">Arthur Koestler</a> also mentions the ability of the higher nervous system to compensate for damages in other parts of the body / psyche as a defining feature of intelligent systems. Somehow I think this happens to be a very Jewish definition of intelligence - wit and intelligence had to compensate for the very disadvantageous situation of discriminations / persecutions / deliberate prescriptions of what you may or may not do for a living. That makes the Wall-E test a very tough test.</p>
<p>Somehow defining intelligence as the ability of a machine to convince with small talk may look like a stereotypically English way of looking at things, probably all these definitions of intelligence mirror some cultural preferences of the person who is framing this definition, in other words, Intelligence is defined by culture X by what culture X values most and what culture X would like the machine to do. So while a definition may be ok for most of the time, still no single definition will ever define the whole experience of being alive / being intelligent.</p>
<p>‘What is Man, what does it mean to be Human’ - the famous <a href="http://en.wikipedia.org/wiki/Sphinx#The_Riddle_of_the_Sphinx">riddle of the mythical Sphinx</a> is now the question that the Robot asks. That question makes sense for a Sphinx - it is half man half animal, similar to the Robot. Some say that’s the reason why we <a href="http://zompist.wordpress.com/2011/06/24/no-singularity-for-me-mater/">will not actually build any Robots</a> , who wants a complex beast like this around? Funny, are the ancient Egyptian, Mesopotamian/Babylonian/Persian, Indian sphinxes asking any existential questions, or is it just it just the Classical Greek copy-sphinx?</p>
<p>Back to our two definitions of intelligence - of course both definitions are equally dumb; both try to define Intelligence as a <a href="http://en.wikipedia.org/wiki/Black_box">black box</a> - try do define the requirements for a intelligent system in terms of the input and expected output only. So in psychology behaviorism falls short when trying to explain things like learning and language, and here we are back to zero …</p>
<p>I have now invested some time to understand <a href="http://openclassroom.stanford.edu/MainFolder/CoursePage.php?course=MachineLearning">this course on machine learning</a>; so far I am not impressed, it’s more of the same parlor tricks. Wikipedia <a href="http://en.wikipedia.org/wiki/Machine_learning">says here</a> - “There are many similarities between machine learning theory and statistical inference, although they use different terms.” , I tend to agree with that.</p>
<h3 id="again-on-the-singularity">Again on the singularity</h3>
<p>Another sphinx like aspect of the machine: with each new stage new questions come up, more questions than answers. What if they manage to build intelligent machines but those fail to be original thinkers? What if real creativity can’t be automated?</p>
<p>That, of course would remind me of the following scene:</p>
<iframe width="560" height="315" src="//www.youtube.com/embed/3MyZTPPgsZA" frameborder="0" allowfullscreen=""></iframe>
work with core dumps from remote site
2011-06-18T00:00:00+00:00
http://mosermichael.github.com//all/projects/2011/06/18/core-dumps
["", "projects"] | <h2 id="work-with-core-dumps-from-remote-site">work with core dumps from remote site</h2>
<p class="publish_date">
18 July 2011
</p>
<p>One thing that happens to C/C** programs is that they crash at customer sites, usually a <a href="http://en.wikipedia.org/wiki/Core_dump">Core dump</a> is left after the crash .</p>
<p>Now it happens that core dump is sent to you from a remote customer site, and joy and wonder, you can’t get a meaningful stack trace out of it. (Usually that is the point to start loud chanting of the <a href="http://www.netfunny.com/rhf/jokes/92q3/coredb.html">Segmentation violation - Core dumped blues</a></p>
<p>Now</p>
<ul>
<li>
<p>One reason may be that the <a href="http://en.wikipedia.org/wiki/Stack_buffer_overflow">stack is smashed</a> in this case you can’t get any information, sorry. (well we <a href="/cstuff/all/projects/2011/06/19/stack-mirror.html">have a solution right here</a> , but that’s a different story …)</p>
</li>
<li>
<p>Another reason is that you have compiled the program with Optimization and frame pointer omission enabled, in this case you won’t see all information either. Well one options is not to do that ( add -fno-frame-pointer-omission) . Well actually <a href="http://www.yosefk.com/"">Yossi Kreinin</a>, with a <a href="http://www.yosefk.com/blog/getting-the-call-stack-without-a-frame-pointer.html">very clever trick</a></p>
</li>
<li>
<p>The third one (very likely) is that the crash occurs in a shared library (libc,pthread, oracle client, etc), and it can’t be traced, because your system has a slightly different versions of this shared library as compared to what you have locally on your machine. Now this situation can be solved, so now it’s scripting time (Perl to the rescue)</p>
</li>
<li>
<p>Another scenario is when a system / third party shared library calls you back, In this case you are bound to see a different stack on your machine, of third party library is different from what you have. An example is C** stack unwinding; when an exception is caught, then GLIBC is calling your code, it is calling object destructors of stack based objects.</p>
</li>
</ul>
<p>The script does the following:</p>
<ul>
<li>
<p>Get path of all dependent shared libraries (parse result of ldd <executable> )</executable></p>
</li>
<li>Copy shared libraries , the core file, the binary file and make an archive out of this</li>
<li>Create a shell script that instructs GDB to use the copied shared libraries instead of what you got on your system. (here goes the trick)</li>
</ul>
<p>Now <a href="https://github.com/MoserMichael/cstuff/blob/master/scripts/gather-libs.pl">here is the script</a> that does it all.</p>
<h3 id="how-to-use-it">how to use it</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./gather-libs.pl -c core.12345 -e exe_file -o out.tbz2
</code></pre></div></div>
<p>Creates the archive out.tbz2 that includes</p>
<ul>
<li>
<p>Core file core.12345</p>
</li>
<li>Executable exe_file and all its dependent shared libraries</li>
<li>The script run.sh that invokes gdb with the copied shared libraries</li>
</ul>
<p>Now once you have copied archive out.tbz2 over to your machine you can open it</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tar xvfj out.bz2
</code></pre></div></div>
<p>Now script run.sh will first check if the executable has any debug information; It happens that applications strip debug information before they ship to a customer, so it is better to check if that happened or not.</p>
<p>If executable has debug information then gdb is run and told to look at the copied shared libraries instead what is installed on your system.
To run with a different core file</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./run.sh -c core.2345
</code></pre></div></div>
<p>To run with a different executable and core</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>,/run.sh -c core.2345 -e another-exe:
</code></pre></div></div>
<h3 id="how-to-tell-gdb-to-look-at-different-set-of-share-libraries">How to tell gdb to look at different set of share libraries.</h3>
<p>Gdb has the following command</p>
<blockquote>
<p>set solib-absolute-path <path></path></p>
</blockquote>
<p>It tells gdb to look for shared libraries from under <path>. Now the archive contains script run.sh ; which writes the following gdb script to a temporary file</path></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>set solib-absolute-prefix .
core-file <core file name>
</code></pre></div></div>
<p>Then it starts gdb with this generated script (-x option of gdb).
Now doing a script that writes another script is an exercise in magic (small magic alas).</p>
<h3 id="limitations">Limitations</h3>
<p>As always, there are situation when this script does not work properly; For example if the application loads shared libraries dynamically with dlopen / dlsym, then those will not be copied into the archive.
(Please tell me that this does not happen often on Linux; Sorry).</p>
<h3 id="interesting-core-dump-trivia">Interesting core dump trivia</h3>
<p>Once upon a time we got a core file that was huge - about three gigabytes large. Can it be understood from the size of the core dump that the application has run out of memory?
Well the answer is a bit tricky,
Let’s remember a few things about <a href="http://en.wikipedia.org/wiki/Virtual_memory">Virtual memory</a> - there is the ‘Virtual memory size’ - for a given process these are all virtual address ranges summed up; it is the number of all reserved virtual addresses; these addresses may or may not be backed up with real memory.
Then there is the ‘Resident set size’ - the number of all physical memory pages that are in use by a process multiplied by page size; (a memory page Is a fixed size unit that backs up a range of virtual memory / is the real memory behind a range of virtual addresses; a page is typically 4096 or 8192 bytes long).
The question is now, can the ‘Resident set size’ be extracted from the core dump?
Now a core dump is just an <a href="http://en.wikipedia.org/wiki/Executable_and_Linkable_Format">ELF format binary file</a> with a special section called “note” - information about the core dump, registers at the time of the crash, etc.
The core dump is created by the Linux kernel; so let’s search for the function that is writing the core dump, the <a href="http://lxr.linux.no/+trees">Linux cross reference project</a> can be used to search the kernel sources, searches here are faster than grep !</p>
<ul>
<li>
<p>First the ELF binary format is described in header <a href="http://lxr.linux.no/#linux+v3.0.4/include/linux/elf.h">include/linux/elf.h</a> ;</p>
</li>
<li>“structure elf32_note”: http://lxr.linux.no/#linux+v3.0.4/include/linux/elf.h#L402 is defined here, it describes the note header that describes the core file. So we should search for platform independent <a href="http://lxr.linux.no/#linux+v3.0.4/include/linux/elf.h#L422">elf_note</a></li>
<li>Now function <a href="http://lxr.linux.no/#linux+v3.0.4/fs/binfmt_elf.c#L1875">elf_core_dump</a> seems to be writing the core dump.</li>
</ul>
<p>The function first writes the executable header and note sections (lines 1903 - 1978) ; then an elf section header is written for each virtual memory address range (1981- 2003) then at line 2019 it starts to get interesting , it writes the actual memory pages into the note ext section.
Line 2019 loops on all virtual memory ranges; for each range the nested loop at line 2026 loops over all pages that make up a memory range. So line 2030 gets the content of a memory page, if the page is available and backed up by physical memory then it is written into the file (2034), if not then line 2039 just extends the file by the size of one range and puts zeroes into it.</p>
<p>So know we know that the core file is equal to the size of the virtual memory. In principle one could search for consecutive zero ranges, but then it is impossible to tell if a range of zeros is a range of virtual addresses that is not backed by physical memory, or a page of physical memory that is all zero.</p>
<p>The loop that writes the memory ranges.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2019 for (vma = first_vma(current, gate_vma); vma != NULL;
2020 vma = next_vma(vma, gate_vma)) {
2021 unsigned long addr;
2022 unsigned long end;
2023
2024 end = vma->vm_start + vma_dump_size(vma, cprm->mm_flags);
2025
2026 for (addr = vma->vm_start; addr < end; addr += PAGE_SIZE) {
2027 struct page *page;
2028 int stop;
2029
2030 page = get_dump_page(addr);
2031 if (page) {
2032 void *kaddr = kmap(page);
2033 stop = ((size += PAGE_SIZE) > cprm->limit) ||
2034 !dump_write(cprm->file, kaddr,
2035 PAGE_SIZE);
2036 kunmap(page);
2037 page_cache_release(page);
2038 } else
2039 stop = !dump_seek(cprm->file, PAGE_SIZE);
2040 if (stop)
2041 goto end_coredump;
2042 }
2043 }
</code></pre></div></div>
<p>The big function</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1868/*
1869 * Actual dumper
1870 *
1871 * This is a two-pass process; first we find the offsets of the bits,
1872 * and then they are actually written out. If we run out of core limit
1873 * we just truncate.
1874 */
1875static int elf_core_dump(struct coredump_params *cprm)
1876{
1877 int has_dumped = 0;
1878 mm_segment_t fs;
1879 int segs;
1880 size_t size = 0;
1881 struct vm_area_struct *vma, *gate_vma;
1882 struct elfhdr *elf = NULL;
1883 loff_t offset = 0, dataoff, foffset;
1884 struct elf_note_info info;
1885 struct elf_phdr *phdr4note = NULL;
1886 struct elf_shdr *shdr4extnum = NULL;
1887 Elf_Half e_phnum;
1888 elf_addr_t e_shoff;
1889
1890 /*
1891 * We no longer stop all VM operations.
1892 *
1893 * This is because those proceses that could possibly change map_count
1894 * or the mmap / vma pages are now blocked in do_exit on current
1895 * finishing this core dump.
1896 *
1897 * Only ptrace can touch these memory addresses, but it doesn't change
1898 * the map_count or the pages allocated. So no possibility of crashing
1899 * exists while dumping the mm->vm_next areas to the core file.
1900 */
1901
1902 /* alloc memory for large data structures: too large to be on stack */
1903 elf = kmalloc(sizeof(*elf), GFP_KERNEL);
1904 if (!elf)
1905 goto out;
1906 /*
1907 * The number of segs are recored into ELF header as 16bit value.
1908 * Please check DEFAULT_MAX_MAP_COUNT definition when you modify here.
1909 */
1910 segs = current->mm->map_count;
1911 segs += elf_core_extra_phdrs();
1912
1913 gate_vma = get_gate_vma(current->mm);
1914 if (gate_vma != NULL)
1915 segs++;
1916
1917 /* for notes section */
1918 segs++;
1919
1920 /* If segs > PN_XNUM(0xffff), then e_phnum overflows. To avoid
1921 * this, kernel supports extended numbering. Have a look at
1922 * include/linux/elf.h for further information. */
1923 e_phnum = segs > PN_XNUM ? PN_XNUM : segs;
1924
1925 /*
1926 * Collect all the non-memory information about the process for the
1927 * notes. This also sets up the file header.
1928 */
1929 if (!fill_note_info(elf, e_phnum, &info, cprm->signr, cprm->regs))
1930 goto cleanup;
1931
1932 has_dumped = 1;
1933 current->flags |= PF_DUMPCORE;
1934
1935 fs = get_fs();
1936 set_fs(KERNEL_DS);
1937
1938 offset += sizeof(*elf); /* Elf header */
1939 offset += segs * sizeof(struct elf_phdr); /* Program headers */
1940 foffset = offset;
1941
1942 /* Write notes phdr entry */
1943 {
1944 size_t sz = get_note_info_size(&info);
1945
1946 sz += elf_coredump_extra_notes_size();
1947
1948 phdr4note = kmalloc(sizeof(*phdr4note), GFP_KERNEL);
1949 if (!phdr4note)
1950 goto end_coredump;
1951
1952 fill_elf_note_phdr(phdr4note, sz, offset);
1953 offset += sz;
1954 }
1955
1956 dataoff = offset = roundup(offset, ELF_EXEC_PAGESIZE);
1957
1958 offset += elf_core_vma_data_size(gate_vma, cprm->mm_flags);
1959 offset += elf_core_extra_data_size();
1960 e_shoff = offset;
1961
1962 if (e_phnum == PN_XNUM) {
1963 shdr4extnum = kmalloc(sizeof(*shdr4extnum), GFP_KERNEL);
1964 if (!shdr4extnum)
1965 goto end_coredump;
1966 fill_extnum_info(elf, shdr4extnum, e_shoff, segs);
1967 }
1968
1969 offset = dataoff;
1970
1971 size += sizeof(*elf);
1972 if (size > cprm->limit || !dump_write(cprm->file, elf, sizeof(*elf)))
1973 goto end_coredump;
1974
1975 size += sizeof(*phdr4note);
1976 if (size > cprm->limit
1977 || !dump_write(cprm->file, phdr4note, sizeof(*phdr4note)))
1978 goto end_coredump;
1979
1980 /* Write program headers for segments dump */
1981 for (vma = first_vma(current, gate_vma); vma != NULL;
1982 vma = next_vma(vma, gate_vma)) {
1983 struct elf_phdr phdr;
1984
1985 phdr.p_type = PT_LOAD;
1986 phdr.p_offset = offset;
1987 phdr.p_vaddr = vma->vm_start;
1988 phdr.p_paddr = 0;
1989 phdr.p_filesz = vma_dump_size(vma, cprm->mm_flags);
1990 phdr.p_memsz = vma->vm_end - vma->vm_start;
1991 offset += phdr.p_filesz;
1992 phdr.p_flags = vma->vm_flags & VM_READ ? PF_R : 0;
1993 if (vma->vm_flags & VM_WRITE)
1994 phdr.p_flags |= PF_W;
1995 if (vma->vm_flags & VM_EXEC)
1996 phdr.p_flags |= PF_X;
1997 phdr.p_align = ELF_EXEC_PAGESIZE;
1998
1999 size += sizeof(phdr);
2000 if (size > cprm->limit
2001 || !dump_write(cprm->file, &phdr, sizeof(phdr)))
2002 goto end_coredump;
2003 }
2004
2005 if (!elf_core_write_extra_phdrs(cprm->file, offset, &size, cprm->limit))
2006 goto end_coredump;
2007
2008 /* write out the notes section */
2009 if (!write_note_info(&info, cprm->file, &foffset))
2010 goto end_coredump;
2011
2012 if (elf_coredump_extra_notes_write(cprm->file, &foffset))
2013 goto end_coredump;
2014
2015 /* Align to page */
2016 if (!dump_seek(cprm->file, dataoff - foffset))
2017 goto end_coredump;
2018
2019 for (vma = first_vma(current, gate_vma); vma != NULL;
2020 vma = next_vma(vma, gate_vma)) {
2021 unsigned long addr;
2022 unsigned long end;
2023
2024 end = vma->vm_start + vma_dump_size(vma, cprm->mm_flags);
2025
2026 for (addr = vma->vm_start; addr < end; addr += PAGE_SIZE) {
2027 struct page *page;
2028 int stop;
2029
2030 page = get_dump_page(addr);
2031 if (page) {
2032 void *kaddr = kmap(page);
2033 stop = ((size += PAGE_SIZE) > cprm->limit) ||
2034 !dump_write(cprm->file, kaddr,
2035 PAGE_SIZE);
2036 kunmap(page);
2037 page_cache_release(page);
2038 } else
2039 stop = !dump_seek(cprm->file, PAGE_SIZE);
2040 if (stop)
2041 goto end_coredump;
2042 }
2043 }
2044
2045 if (!elf_core_write_extra_data(cprm->file, &size, cprm->limit))
2046 goto end_coredump;
2047
2048 if (e_phnum == PN_XNUM) {
2049 size += sizeof(*shdr4extnum);
2050 if (size > cprm->limit
2051 || !dump_write(cprm->file, shdr4extnum,
2052 sizeof(*shdr4extnum)))
2053 goto end_coredump;
2054 }
2055
2056end_coredump:
2057 set_fs(fs);
2058
2059cleanup:
2060 free_note_info(&info);
2061 kfree(shdr4extnum);
2062 kfree(phdr4note);
2063 kfree(elf);
2064out:
2065 return has_dumped;
2066}
2067
</code></pre></div></div>
First Post
2011-06-17T00:00:00+00:00
http://mosermichael.github.com//all/ramblings/2011/06/17/About-me-First-Post
["", "ramblings"] | <h2 id="first-post">First Post</h2>
<p class="publish_date">
17 July 2011
</p>
<p>My name is Michael Moser; I used to have a home page <a href="http://web.archive.org/web/20031228111524/http://www.michaelmoser.org/">long long time ago</a> Seems ages now. Ever since that time a lot of things changed, so this is my somewhat belated comeback as WEB 2.0. Now any page/content here on this site may change without warning, just like Wikipedia ;-)</p>
<p>The posts are formatted in wiki-like markup language <a href="http://redcloth.org">Textile</a> , the site is generated by <a href="https://github.com/mojombo/jekyll">Jekyll</a> and the web site template I modified from <a href="http://danhixon.github.com">Dan Hixon’s blog</a> . <a href="http://github.com">Github</a> is hosting this stuff. Free comments/discussion boards are hosted by <a href="http://disqus.com/">Disqus</a> . Many thanks to everyone involved.</p>
<p>Also special thanks to the <a href="http://en.wikipedia.org/wiki/Vint_Cerf">two</a> <a href="http://en.wikipedia.org/wiki/Bob_Kahn">inventors</a> and developers of TCP/IP; The <a href="http://en.wikipedia.org/wiki/Tim_Berners-Lee">inventor</a> of HTTP, and the many thousand workers and developers that deal with the wondrous world of <a href="http://www.rfc-editor.org/rfc.html">IP routing and switching</a>. The result of their common toil may turn out to be as important as the invention of the <a href="http://en.wikipedia.org/wiki/Johannes_Gutenberg">printing press</a></p>
<p>(Currrent Music : <a href="http://www.youtube.com/watch?v=OPHLZPlXgWY&feature=related">Buffalo Springstreen - Uno Mundo</a> )</p>
<iframe width="420" height="345" src="http://www.youtube.com/embed/XED-lhXKYvc" frameborder="0" allowfullscreen=""></iframe>
The make file to end all make files
2011-06-17T00:00:00+00:00
http://mosermichael.github.com//all/projects/2011/06/17/make-system
["", "projects"] | <h2 id="the-make-file-to-end-all-make-files">The make file to end all make files</h2>
<p class="publish_date">
17 July 2011
</p>
<p>One of the things I do for living is fiddling with make files with <a href="http://www.gnu.org/software/make/">GNU Make</a> . Usually this leaves me with the vague impression that there must be a better way to do this and that. And so it’s time to do it right, even if it is only for my own stuff.
So each piece of software should start with a list of requirements, here is my list for the make file</p>
<ul>
<li>Support make/test/install/clean/cleanall targets.</li>
<li>Make script runs on multiple operating systems (Linux and Cygwin for a start). It is possible to override options per OS / OS version / processor type.</li>
<li>All build results (objects, executables, libraries) are placed under one root directory. Root directory name contains OS name + OS version + processor type.</li>
<li>Multiple targets of different type can be declared in one make file; if one has forgotten declaration required for target, a human readable error is displayed.</li>
<li>The object files of each target are placed in separate object directory, this way we can set separate compiler option for intersecting source file sets.</li>
<li>A build target can be of type exe/shared library/static library.</li>
<li>A build target either includes subset of files or all files in directory.</li>
<li>A build target can add its specific compiler/linker settings.</li>
<li>Track source file dependencies; use trick that avoids reparsing of make file.</li>
<li>A target can declare dependent libraries;</li>
<li>Each make file has optional pre build and post build steps that can do code generation for instance.</li>
<li>Display banner on building of each target; (what is the target, type of target + directory)</li>
<li>Ability to add compiler and linker options from environment to each target (environment variables EXTERNAL_CFLAGS , EXTERNAL_CXXFLAGS , EXTERNAL_LDFLAGS ).</li>
<li>Test target: all test programs and test libraries are put into separate directory $(BIN_ROOT_DIR)/test</li>
<li>Try to minimize number of shells invoked; try to minimize number of make file rereads/re reads. This is important for Cygwin support + compilation speed.</li>
<li>Like all software, the build script has a set of tests.</li>
</ul>
<p>The make system magic works by reusing rule definition files all over each Makefile.
Each directory that builds a target has its own Makefile, this file has to includes the skeleton rule file.</p>
<p>The skeleton rule files are placed in the root directory and those are</p>
<ul>
<li><a href="https://github.com/MoserMichael/cstuff/blob/master/rules.make">rules.make</a></li>
<li><a href="https://github.com/MoserMichael/cstuff/blob/master/rules-GNU-Linux.make">rules-GNU-Linux.make</a></li>
<li><a href="https://github.com/MoserMichael/cstuff/blob/master/rules-Cygwin.make">rules-Cygwin.make</a>
You also get it with my <a href="https://github.com/MoserMichael/cstuff">GIT repository for stuff in c</a></li>
</ul>
<h3 id="basic-usage--defining-build-targets">Basic usage + defining build targets</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1: TOPDIR=../..
2:
3: # - declare build targets. (built with make)
4: TARGETS:=shlib slibuser
5:
6: # - slib target is a static library -
7: shlib_TYPE=lib
8: shlib_SRC=slib.c
9:
10:
11: # - slibuser target is a executable using slib -
12: slibuser_TYPE=exe
13: slibuser_SRC=slibuser.c slibuser2.c slibuser3.c
14: slibuser_LIBS=shlib
15:
16: include $(TOPDIR)/rules.make
</code></pre></div></div>
<p>Each Makefile has the TOPDIR variable that points to the root directory, in the root directory we have the skeleton rule files. Line 16 includes the rule.make - the skeleton rule file.
The List of build targets is declared in line 4; the TARGETS variable lists the name of each target; in this case targets shlib and slibuser are declared.
Each build target defines its type
slibuser_TYPE=exe
This means that build target slibuser builds a executable.
slibuser_TYPE=shlib
This means that build target slibuser builds a shared library.
slibuser_TYPE=lib
This means that build target slibuser builds a static link library.
Each target is built from sources, so either it lists all the source files:
slibuser_SRC=slibuser.c slibuser2.c slibuser3.c
Says that build target slibuser is built from the source files slibuser.c slibuser2.c and slibuser3.c</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1: TOPDIR=../..
2:
3: # - declare build targets. (built with make)
4: TARGETS:=shlib
5:
6: shlib_TYPE=shlib
7: shlib_SRC_EXTENSION=c
8:
9
10: include $(TOPDIR)/rules.make
</code></pre></div></div>
<p>shlib_SRC_EXTENSION=c
This declaration says that the build target shlib is built from all files with file extension *.c in the current directory.</p>
<h3 id="build-results">build results</h3>
<p>The build result directory is a sub directory of the root directory (a sub directory of $(TOPDIR); The name of the build result directory is made up of the following components</p>
<ul>
<li>Operating system name (based on uname <del>o, sanitized by replacing / ( ) characters with</del> )</li>
<li>Kernel release (based on uname <del>r, sanitized by replacing / ( ) characters with</del>)</li>
<li>Machine architecture (based on uname <del>m, sanitized by replacing / ( ) characters with</del>)
For my current system the name is GNU-Linux-2.6.38-8-generic-i686 ;</li>
</ul>
<p>For each executable build target the result is placed into $(BUILD_RESULT_DIRECTORY)/bin, static or shared libraries are placed into $(BUILD_RESULT_DIRECTORY)/lib
Test programs and libraries are placed into $(BUILD_RESULT_DIRECTORY)/test</p>
<h3 id="adding-specific-compiler--linker-options-to-a-project">adding specific compiler / linker options to a project.</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1: TOPDIR=../..
2:
3: # - declare build targets. (built with make)
4: TARGETS:=shlib
5:
6: # - slib target is a static library -
7: shlib_TYPE=shlib
8: shlib_SRC_EXTENSION=c
9: shlib_CFLAGS=-finstrument-functions -O2 -fno-omit-frame-pointer
10: shlib_LIBS=pthread cutils
11:
12: include $(TOPDIR)/rules.make
</code></pre></div></div>
<p>shlib_CFLAGS=-finstrument-functions
This declares that compiler option -finstrument-functions is added specifically to target shlib; the option is specifically added to compilation of C files.
In order to add options to C** command line one should have set shlib_CXXFLAGS variable.
shlib_LIBS=pthread cutils
This declares that the shlib target is linked with libraries pthread and cutils; the -lpthread and -lcutils is added to the linker options for this target.</p>
<h3 id="recursing-into-subdirectories">Recursing into subdirectories</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1: TOPDIR=../..
2:
3: PREBUILD_SUBDIRS=dir-a dir-b dir-c
4:
5: # - declare build targets. (built with make)
6: TARGETS:=shlib
7:
8: # - slib target is a static library -
9: shlib_TYPE=shlib
10: shlib_SRC_EXTENSION=c
11: shlib_CFLAGS=-finstrument-functions -O2 -fno-omit-frame-pointer
12: shlib_LIBS=pthread cutils
13:
14: include $(TOPDIR)/rules.make
</code></pre></div></div>
<p>PREBUILD_SUBDIRS declares that prior to building target shlib, the make process would recurse into sub directories and invoke Makeefile in directories dir-a dir-b and dir-c
POSTBUILD_SUBDIRS declares that after building target shlib, the make process would recurse into sub directories and invoke Makeefile in directories dir-a dir-b and dir-c</p>
<h3 id="specifying-test-targets-invoked-by-make-test">Specifying test targets invoked by ‘make test’</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1: TOPDIR=../..
2: TESTS=cutilstest
3: cutilstest_TYPE=exe
4: cutilstest_SRC_EXTENSION=c
5: cutilstest_LIBS=cutils vtest
6: include $(TOPDIR)/rules.make
</code></pre></div></div>
<p>The TESTS variable lists name of test programs; these programs compiled and run by ‘make test’ command.</p>
<h3 id="adding-custom-pre-build-step">Adding custom pre build step</h3>
<p>Each target can execute a pre build step - a script/program that is run before making any source files. This can be used either to</p>
<ul>
<li>Generation of source code</li>
<li>A self configuration script that tests current environment and writes a header file that defines all sort of options.
The pre build test program receives the names of all source files of this target as command line parameters.
A pre build step is defined by the following declaration
<target name="">\_PREBUILD=<prebuild script="" path="">
h3. Adding a custom post build step
Each target can execute a post build step - a script/program that is run after making any source files. This can be used either to
</prebuild></target>
</li>
<li>custom packaging</li>
<li>Instrument the binary of the executable</li>
<li>Run the target binary yas part of complicated test script.</li>
</ul>
<p>The first argument of the post build script is always the path name of the file that is built by the target.
A post build step is defined by the following declaration</p>
<target name="">\_POSTBUILD=<postbuild script="" path="">
### Invocation of make
#### make
Running make will build all targets in debug mode;
make
This next command will build the code with full optimization, and with debug symbols; one issue is that executables and libraries are put in the same place for both debug and release builds; However the object files for release and debug builds are put in separate directories and do not intersect.
make MODE=release
This next command willl build build the code with less optimization (on Linux it uses gcc and -O2).
make MODE=release2
We can also specify the desired type of executable produced.
make ARCH=x86\_64
This will build a 64 bit executable (on the Intel platform that is)
make ARCH=i686
This will build a 32 executable.(on the Intel platform that is)
Also there are some other nice tricks in store, tricks that are sometimes very useful during development
make CPP\_LISTING=1
Now the source file directory will also have a file with the [C preprocessor output](http://gcc.gnu.org/onlinedocs/cpp/Preprocessor-Output.html) for each compiled file.
For example the source file foo.c if compiled by target bar will have the file foo.bar.i with preprocessor output.
The preprocessor output can be priceless when you have to fix compiler errors that involve macros for instance.
The preprocessor listing is created prior to compilation of a source file, as it is usually needed when we try to fix a weird compiler error
Also the following invocation will produce the assembly listing for each compiled file.
make ASM\_LISTING=1
For example the source file foo.c if compiled by target bar will have the file foo.bar.S with assembly listing.
The assembly listing can be priceless if you have to understand what happens with advanced optimization modes, for instance.
The assembly listing is created after compilation of a source file, when we are sure that the stuff actually compiles.
The downside is that the listing files are not deleted by running 'make clean ' or 'make cleanall ' you have to remove them manually.
#### make test
Will first build all targets and then build all test targets
You can specify a prefix command that will run the test targets; for instance
make test TEST\_TOOL=valgrind
Here all test programs will be checked by [valgrind](http://valgrind.org/)
Running valgrind on unit tests is something that one definitely wants to do - on the one hand a unit test is supposed to be short, so that valgrind runtime overhead will not be that significant, on the other hand unit tests are supposed to cover a lot of functionality - so this tool will help a lot here.
Alternatively the following command will not run any tests at all;
make test TEST\_TOOL=true
Sometimes you really want to skip running tests, like when running a test is done by a post built step, or if you build with a cross compiler, for instance.
#### make clean
Deletes all object files and executable targets
#### make cleanall
Removes the whole build result directory tree; everything that is built is hereby removed in one simple step. Every file, with the noted exception of stuff that has been generated by scripts invoked prior to build process (pre-build step) or after build (post-build steps).
#### make install
Copies all built libraries into $(PREFIX)/lib and all built executables into $(PREFIX)/bin
The default prefix is /usr/local
make install INSTALL\_PREFIX=/opt/mystuff
Install to the very different prfix /opt/mystuff
### Limitation of make system
All this magic works with gmake 3.81 and upwards. It uses the built-in $(eval) and $(call) macro facilities; and these do not work properly on all versions of make.
Yes, I know that this statement is killing the punch line;-)
That seems to be a general rule; whenever one is using more sophisticated features of software product X the more you do rely on a specific version of the X product;
So the make file to end all make files does not work on older version of gmake.
Another major limitation is that I did not have to deal with the problem of cross builds; so there is no generic mechanism for dealing with cross build; I will have to devise a way that is compatible with other make systems. Automake tools do a lot in the respect; I still find it easier to run my own stuff;
### Debugging the make file
make SHOW\_RULES=1
This command will dump to the screen most macro expansions that are done behind the scene. It does not show all built rules though.
I think any macro system should have such an option built in - to dump the output to screen, but again, life is life.
</postbuild></target>
PERL my getopt command line parser
2011-06-17T00:00:00+00:00
http://mosermichael.github.com//all/blog/2011/06/17/my-perl-getop
["", "blog"] | <h2 id="perl-my-getopt-command-line-parser">PERL my getopt command line parser</h2>
<p class="publish_date">
17 July 2011
</p>
<p>Usually I want my PERL scripts to work on all machines that have PERL installed.
Now a script usually parses the command line, in perl you have <a href="http://perldoc.perl.org/Getopt/Std.html">Getopt::Std</a> to does that for you.
Now this command line parser is in a core module, however this does not mean that it is available on each PERL installation, for example the default installation of RedHat Enterprise Server installs PERL without the core modules; so we can’t use Getopt::Std for parsing command lines here.</p>
<p>Fortunately we can roll our own in 30 lines, so there is no problem in copying this function over into other scripts; as we all know copying code is an awful engineering practice, that should be punished by decapitation of the offender (in most cases the offender is acting without a head to begin with, so this measure becomes impractical ;-)
That does not mean that we should <a href="http://xkcd.com/292/">Screw good practices</a> - punishment is never <a href="http://xkcd.com/292/">far behind</a></p>
<p>Here we can justify copy-paste by our desperate goal of running a PERL script on all installations.
Now <a href="https://github.com/MoserMichael/cstuff/blob/master/scripts/m-getop.pl">here is the script</a>
As you can see, the getopt function receives an array, where each entry describes an option, this simplifies the function to the extent that we don’t have to parse the argument specification.
Otherwise it is all quite similar to the original <a href="http://perldoc.perl.org/Getopt/Std.html">Getopt::Std</a></p>
<h3 id="almost-ten-years-later">almost ten years later.</h3>
<p>Nowadays perl5 is not quite used as a scripting language; in a corporate environment that is something that is frowned upon. Meanwhile they even added <a href="https://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash">hash tables to bash</a>, so that is supposed to be the way to go. (doing tasks that shell out to programs and do some text processing is less practical in python)</p>
<p>Now those who didn’t like perl5 got it much worse - bash is such an awful language…. (at least it has tracing in the form of set-x)</p>
The license of this stuff / Why use C ?
2011-06-16T00:00:00+00:00
http://mosermichael.github.com//all/projects/2011/06/16/license
["", "projects"] | <h2 id="the-license-of-this-stuff--why-use-c-">The license of this stuff / Why use C ?</h2>
<p class="publish_date">
17 July 2011
</p>
<p>.. is the “ 3-clause BSD also known as the <a href="https://en.wikipedia.org/wiki/BSD_licenses#3-clause">New BSD License</a>
Which gives you license to do as you please with this stuff, except that you may not sue me for any damage and you have to credit me for using stuff.</p>
<h3 id="why-c">Why C</h3>
<p>These projects are mostly written in C with classes / Abstract data types. Here are my reasons to prefer C over C++* or Java.</p>
<p>Higher order languages like C++ and Java have a much more expressive syntax. Now with these capabilities comes an abundance of stylistic choices that divert attention from the real goal of writing a program.</p>
<ul>
<li>With C++ one has to make a set of decisions; is it wise to use C++ exceptions / RTTI or not? Will one use the STL library? What about C++ exceptions or not ? What about BOOST, or rather which subset of BOOST? What is your position on template metaprogramming (I strongly dislike this kind of abuse) ? All these choices are normally codified into a style guide, for example the <a href="http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml">Google C++ style guide</a> is a very thorough document and I largely agree with what it says; however all these decision points taken together tend to restrict you with regards to third party libraries and tools that you can interact with; so they in fact restrict your freedom of choice. Now open source is mainly about freedom of choice, both for users and for developers, so there is some conflict here.</li>
</ul>
<!-- -->
<ul>
<li>In Java you have much less choice in terms of styles and programming paradigms; one should generally stick with how the JDK does it. However with Java you have an abundance of framework for any occasion; Choice of adopted framework has a strong lock in affect (will you use SPRING or OSGI or whatever kind of Beans? Which subset of the <a href="http://jakarta.apache.org/">Jakarta</a> alphabet soup is in use? All these choices usually come with a lot of internal politics, programmers tend to invest and band around technological choices.</li>
</ul>
<!-- -->
<ul>
<li>With C++ and Java it is very easy to declare classes; now there are infinite ways how one can possible decompose a system into classes; and you can spend an infinite amount of time on that issue alone; All this of course distracts from the task of writing the actual program. With C with classes one is generally limited to object aggregation and polymorphism; interestingly the “Design pattern book”: advocates preference of object aggregation over inheritance. So somehow less choice sometimes results in more freedom.</li>
</ul>
<!-- -->
<ul>
<li>Interesting point is that in recent years there is stated a trend away from OO design processes towards Test driven / Agile methodologies;</li>
</ul>
<!-- -->
<ul>
<li>Also <a href="http://www.realworldtech.com/forums/index.cfm?action=detail&id=110618&threadid=110549&roomid=2">Linus Thorvalds says</a> that with C one usually does not have to wonder what a line of code does; as opposed to higher language features that are of course expressive, but of course they are supposed to ‘hide complexity’ so go figure what the thing does.</li>
</ul>
<!-- -->
<ul>
<li>An obligatory reference to the <a href="http://yosefk.com/c++fqa/">C+ FQA</a> is hereby made.</li>
</ul>
<p>The point is that I still do C++ and Java for paid work, as best as I can; but all the fluff and politics around it is; well, really no fun. Also please note that most programs that matter are written in C, the Linux kernel, the gcc toolchain, most scripting language interpreters - all in C;</p>
<p>While C++ may be best suited for companies that build large monolithic applications, it is not the best choice for your garage open source project.</p>
<p>Still C++ is a good language for comercial projects, on condition that one really needs to utilise the computing machine to its maximum. The language creates a common vocabulary for such things as collection classes and object oriented design with patterns, things that are very important when trying to communicate the design of a project to a new programmer, just added to the team.</p>
<p>Still Java may be the best language, on condition that one does not strive to utilize the computing machine to its very maximum, but one still cares for good performance. Its development tools allow for great productivity, automated memory management may still be the best improvement in programmer productivity, maybe even by an order of magnitude. One can build very tall abstractions in java, however one still may need the help of the occasional hackers, those that care for ‘low level details’; one needs them for example when sorting out things like reference leaks, and helpng out if things don’t quite work ;-)</p>
<h3 id="ten-years-later">ten years later</h3>
<p>actually most server side software runs on the cloud, and they can use the regular GPL as well without a problem. I think that this is a reversal of great importance.
as for programming in c, i wouldn’t use that in this day and age. C++ got a bit more sophisticated and nowadays it is much more practical to deal with templates for metaprogramming rather than with C macros.</p>